Generate Engine Meta Data

[ehynd]

Is there an existing issue for this?

• I confirm that I have not found an existing or similar issue.

Description

Using LLM, generate meta data for empty fields within an engine.
Should not overwrite fields that contain data

Ex: If description and tags are empty, generate both. If only tags is empty, only generate tags.

Relevant Feature or Code

No response

Requirements (what)

• Generate meta data for empty fields
• User should have some way to select what info is being sent to LLM for generation. Ex: give option for engine schema, sample rows, typing in additional details etc. User should always know what is being sent so data is not being leaked unknowingly.

[shubhammahure]

How to Test

You can test in run pixel(Notebook cell) call
just replace your engineid and model id and provide options listed below(engine wise)
GenerateEngineMetadata(engine=["engineid"], model=["modelid"], metaKeys=[["description","tag"]], storagePath=["storagepath"], filePath=['filepath'], options=[{"includeSchema":true,"includeVectorFileNames":false,"includeVectorChunks":true,"includeStorageFileNames":true,"includeModelSmssInfo":true, "includeFunctionSmssInfo":2, "includeStorageFileContent":true }]);

Example for StorageEngine
GenerateEngineMetadata(engine=["ed23348f-842c-417d-...."], model=["4801422a-5c62-421e-a00c-05c6...."], metaKeys=[["description","tag"]], storagePath=["/abcd3"], filePath=['/shubh'], options=[{"includeStorageFileNames":true,"includeStorageFileContent":true }]);

Note: The storagePath and filePath keys are only supported for StorageEngine

Replace options according to your engine:

Supported options:

1) Database

• includeSchema (boolean):
  Include database schema details such as tables and columns.
• tableSchemaLimit (number):
  Max tables to include (default: 5)
• columnSchemaLimit (number):
  Max columns per table (default: 10)

2) Vectors

• includeVectorFileNames (boolean):
  Include vector document file names.
• vectorFileLimit (number):
  Maximum number of vector file names to include (default: 5).
• includeVectorChunks (boolean):
  Include sample text chunks from vector documents.
• vectorChunkLimit (number):
  Maximum number of vector text chunks to include (default: 3).

3) Storages

• includeStorageFileNames (boolean):
  Include storage file names.
• storageFileNameLimit (number):
  Maximum number of storage file names to include (default: 5).
• includeStorageFileContent (boolean):
  Copy files locally and read content snippets for context.
• storageFileLimit (number):
  Max number of files to read content from (default: 3).
• storageCharLimit (number):
  Max characters to read per file (default: 500).

4) Models

• includeModelSmssInfo (boolean):
  Include model smss file content.

5) Functions

• includeFunctionSmssInfo (boolean):
  Include function smss file content.

6) User-specific options

• useExistingDescription (boolean):
  Refine and enhance an existing description instead of generating a new one.
• additionalContext (string):
  Extra user-provided context to guide metadata generation.
• tone (string):
  Writing tone for the generated description.
  Supported values: neutral | business | scientific (default: neutral).****