新增阿里通义万相wan视频模型支持#2139
Conversation
WalkthroughAdds Ali video generation channel support by introducing a new TaskAdaptor implementation, refactoring form/JSON unmarshalling in common utilities, extending relay integration structures, and updating video proxy handling to support Ali-specific videoURL sources. Changes
Sequence DiagramsequenceDiagram
participant Client
participant Controller as video_proxy
participant Relay as relay/adaptor
participant Ali as ali/adaptor
participant AliAPI as Ali API
Client->>Controller: Video request (Ali channel)
Controller->>Relay: GetTaskAdaptor(ChannelTypeAli)
Relay-->>Controller: taskali.TaskAdaptor
Controller->>Ali: ValidateRequestAndSetAction()
Ali->>Ali: Parse JSON body, validate model/prompt
Ali-->>Controller: ✓ Valid
Controller->>Ali: BuildRequestURL()
Ali-->>Controller: Ali API endpoint
Controller->>Ali: BuildRequestHeader()
Ali-->>Controller: Headers (Authorization, Content-Type)
Controller->>Ali: BuildRequestBody()
Ali->>Ali: convertToAliRequest()
Ali-->>Controller: JSON request body
Controller->>Ali: DoRequest()
Ali->>AliAPI: POST video generation request
AliAPI-->>Ali: Response (task_id, output, etc.)
Ali-->>Controller: HTTP Response
Controller->>Ali: DoResponse()
Ali->>Ali: validateErrorCode, convertToOpenAIVideo()
Ali-->>Controller: taskID, taskData, taskErr
Controller-->>Client: Video generation response (OpenAI format)
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes
Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
relay/common/relay_utils.go (1)
110-176: Function name is misleading AND missing data URL validation on InputReference.
ValidateMultipartDirectnow exclusively handles JSON requests for Ali and Sora channels (verified by call sites). However, two issues need correction:
Semantic mismatch: Function name suggests multipart/form-data handling but implementation is JSON-only. Consider renaming to
ValidateTextToImageRequestorValidateDirectImageGenRequest.Missing data URL validation: Lines 128–130 map
InputReferencedirectly toImagesarray without checking fordata:prefix. This is inconsistent with the pattern used inrelay/relay_task.go:354andrelay/channel/task/vertex/adaptor.go:322, where data URLs are explicitly prevented. Add validation:if req.InputReference != "" { if !strings.HasPrefix(req.InputReference, "data:") { req.Images = []string{req.InputReference} } }
🧹 Nitpick comments (2)
relay/common/relay_info.go (1)
509-539: Consider logging metadata parsing errors for debugging.The flexible metadata parsing (supporting both stringified JSON and direct objects) is well-implemented. However, parsing errors are silently ignored, which could make debugging difficult if clients send malformed metadata.
Consider adding debug logging when metadata parsing fails:
func (t *TaskSubmitReq) UnmarshalJSON(data []byte) error { type Alias TaskSubmitReq aux := &struct { Metadata json.RawMessage `json:"metadata,omitempty"` *Alias }{ Alias: (*Alias)(t), } if err := common.Unmarshal(data, &aux); err != nil { return err } if len(aux.Metadata) > 0 { var metadataStr string if err := common.Unmarshal(aux.Metadata, &metadataStr); err == nil && metadataStr != "" { var metadataObj map[string]interface{} if err := common.Unmarshal([]byte(metadataStr), &metadataObj); err == nil { t.Metadata = metadataObj return nil + } else { + common.LogDebug(c.Request.Context(), fmt.Sprintf("Failed to parse metadata string: %v", err)) } } var metadataObj map[string]interface{} if err := common.Unmarshal(aux.Metadata, &metadataObj); err == nil { t.Metadata = metadataObj + } else { + common.LogDebug(c.Request.Context(), fmt.Sprintf("Failed to parse metadata object: %v", err)) } } return nil }Note: This assumes a debug logging function exists. Adjust based on your logging infrastructure.
relay/channel/task/ali/adaptor.go (1)
149-198: Review metadata extraction approach and consider error handling.The conversion logic is generally solid, but there are a few areas to consider:
Metadata extraction (lines 191-195): The marshal-then-unmarshal approach to extract metadata into the request is clever, but it silently ignores errors and could potentially overwrite previously set default values (like
PromptExtendandWatermark). This might be intentional for allowing user overrides, but consider whether this behavior should be documented or if certain fields should be protected from metadata overrides.Images array handling (line 154): The code maps
InputReferencetoImgURL, but theTaskSubmitReqalso has anImages[]array. From the context inrelay_utils.go,InputReferenceis converted toImages[]. Should this code handle theImagesarray directly, or is the singleInputReferencesufficient for Ali's API?Consider making the metadata extraction more explicit:
// 从 metadata 中提取额外参数 if req.Metadata != nil { - if metadataBytes, err := common.Marshal(req.Metadata); err == nil { - _ = common.Unmarshal(metadataBytes, aliReq) + var aliMetadata AliMetadata + if metadataBytes, err := common.Marshal(req.Metadata); err == nil { + if err := common.Unmarshal(metadataBytes, &aliMetadata); err == nil { + // Explicitly map metadata fields to avoid unintended overwrites + if aliMetadata.AudioURL != "" { + aliReq.Input.AudioURL = aliMetadata.AudioURL + } + // ... map other fields explicitly + } } }Or, document the current behavior if metadata overrides are intentional.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
common/gin.go(4 hunks)controller/video_proxy.go(1 hunks)relay/channel/task/ali/adaptor.go(1 hunks)relay/channel/task/ali/constants.go(1 hunks)relay/common/relay_info.go(3 hunks)relay/common/relay_utils.go(1 hunks)relay/relay_adaptor.go(2 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-08-26T09:59:00.337Z
Learnt from: Sh1n3zZ
Repo: QuantumNous/new-api PR: 1659
File: relay/relay_task.go:285-305
Timestamp: 2025-08-26T09:59:00.337Z
Learning: In controller/task_video.go, data: URLs (containing base64 encoded video data) are prevented from being stored in task.FailReason by checking if the URL starts with "data:" before assignment. This same pattern should be applied consistently across the codebase.
Applied to files:
controller/video_proxy.go
🧬 Code graph analysis (6)
relay/relay_adaptor.go (3)
constant/channel.go (1)
ChannelTypeAli(21-21)relay/channel/task/ali/adaptor.go (1)
TaskAdaptor(105-109)relay/channel/adapter.go (1)
TaskAdaptor(34-53)
relay/common/relay_info.go (1)
common/json.go (1)
Unmarshal(9-11)
controller/video_proxy.go (1)
constant/channel.go (2)
ChannelTypeGemini(28-28)ChannelTypeAli(21-21)
common/gin.go (1)
common/json.go (2)
Unmarshal(9-11)Marshal(21-23)
relay/channel/task/ali/adaptor.go (11)
model/task.go (5)
TaskStatus(13-13)TaskStatusQueued(35-35)TaskStatusInProgress(36-36)TaskStatusSuccess(38-38)TaskStatusFailure(37-37)relay/common/relay_info.go (3)
RelayInfo(76-123)TaskSubmitReq(488-499)TaskInfo(541-550)relay/common/relay_utils.go (1)
ValidateMultipartDirect(110-176)common/gin.go (1)
UnmarshalBodyReusable(33-58)common/json.go (2)
Marshal(21-23)Unmarshal(9-11)relay/channel/api_request.go (1)
DoTaskApiRequest(301-323)service/error.go (1)
TaskErrorWrapper(140-157)dto/openai_video.go (7)
NewOpenAIVideo(43-47)OpenAIVideoError(49-52)VideoStatusQueued(10-10)VideoStatusInProgress(11-11)VideoStatusCompleted(12-12)VideoStatusFailed(13-13)VideoStatusUnknown(9-9)common/utils.go (1)
GetTimestamp(266-268)service/http_client.go (1)
GetHttpClient(49-51)relay/channel/task/ali/constants.go (2)
ModelList(3-9)ChannelName(11-11)
relay/common/relay_utils.go (2)
relay/common/relay_info.go (1)
TaskSubmitReq(488-499)common/gin.go (1)
UnmarshalBodyReusable(33-58)
🔇 Additional comments (15)
common/gin.go (2)
43-47: LGTM: Corrected pointer semantics.The change from
&vtovin theUnmarshalcalls is correct. Sincevis already a pointer type (passed by callers as&req), taking its address again (&v) would create a pointer-to-pointer, which is incorrect. The updated code properly passes the pointer directly toUnmarshal.
147-159: LGTM: Centralized form-to-JSON conversion.The new
processFormMaphelper eliminates code duplication by centralizing the map-to-struct conversion logic via JSON marshaling/unmarshaling, improving maintainability.relay/relay_adaptor.go (1)
31-31: LGTM: Ali task adaptor integration.The import and switch case addition follow the established pattern for channel adaptor registration and are correctly implemented.
Also applies to: 137-138
relay/channel/task/ali/constants.go (1)
3-11: LGTM: Ali channel constants.The model list and channel name declarations are clear and align with the PR objectives. The inline comments (in Chinese) provide helpful context for each model variant.
relay/common/relay_utils.go (1)
128-130: LGTM: InputReference to Images mapping.The mapping of
InputReferenceto theImagesarray (when provided) correctly supports the new Ali video generation flow where a single reference input can be specified.controller/video_proxy.go (1)
94-105: No issues found. Data: URL filtering is already in place across the codebase.The concern about data: URLs being stored in
task.FailReasonis already addressed. The relay_task.go:356-363 applies data: URL filtering—if the URL starts with "data:", it's skipped; otherwise, it stores in FailReason. Ali's VideoURL flows through taskResult.Url and is filtered before storage, consistent with the pattern in controller/task_video.go. The video_proxy.go code is safe.While FailReason semantically implies error information rather than successful URLs, this is an established pattern in the codebase and does not introduce security or functional risks.
relay/common/relay_info.go (2)
4-4: LGTM!The
encoding/jsonimport is appropriately added to support the customUnmarshalJSONmethod for flexible metadata parsing.
488-499: LGTM!The new fields
Seconds,InputReference, andMetadataappropriately extendTaskSubmitReqto support video generation workflows. The use ofomitemptykeeps these fields optional, which is suitable for backward compatibility.relay/channel/task/ali/adaptor.go (7)
25-100: LGTM!The Ali API request/response structures are well-defined with appropriate JSON tags and comprehensive field coverage. The use of pointers for optional fields in
AliVideoParameters(likeAudio *bool) correctly distinguishes between unset and false values.
111-120: Initialization and validation logic looks correct.The adaptor properly initializes with channel metadata and delegates validation to the common utility. Note: The function name
ValidateMultipartDirectmight seem contradictory with the comment stating Ali uses JSON format, but from the codebase context, this function actually handles JSON viaUnmarshalBodyReusable.
122-132: LGTM!The request URL and headers are correctly configured for Ali's async video generation API. The
X-DashScope-Async: enableheader is properly set to enable asynchronous task processing.
134-148: LGTM!The request body building correctly unmarshals the task request and delegates to the conversion logic.
200-243: LGTM!The response handling properly unmarshals the Ali API response, validates the task ID, and converts to OpenAI-compatible format. Error cases are appropriately handled with informative error messages.
246-305: LGTM!The task fetching and result parsing logic correctly handles Ali's async task API. The status mapping comprehensively covers all Ali task states (PENDING, RUNNING, SUCCEEDED, FAILED, CANCELED, UNKNOWN) and appropriately translates them to internal statuses.
264-347: LGTM!The helper methods are well-implemented:
GetModelList()andGetChannelName()return appropriate constantsConvertToOpenAIVideo()correctly transforms the stored task data into OpenAI-compatible format, with the video URL properly set in metadata (line 321)convertAliStatus()provides consistent status mapping throughout the adaptor
|
在另一个pr把你的commit带过去了 |
2 participants
官方文档: https://help.aliyun.com/zh/model-studio/image-to-video-api-reference?spm=a2c4g.11186623.help-menu-2400256.d_2_3_0.2fd96340ai7bZG&scm=20140722.H_2867393._.OR_help-T_cn~zh-V_1
3. 支持openAI sdk 视频生成请求
4. 请求格式:
厂商额外参数: 以json格式放到metadata
5. 请求示例:
6.
curl http://localhost:3000/v1/videos/f06a6f77-e5ab-4eff-8549-3e3eeff4e6ad
8.任务示例: