Fn to FC

python module to convert a given Fn into FC automatically

Example functions

For our first MVP, response generation mostly concern with GPT models answering generic questions and using a single tool for capturing and extracting information from a Youtube Livestream.

import glob
from llmcam.ytlive import YTLive, NHsta
from llmcam.gpt4v import ask_gpt4v

source

capture_youtube_live_frame_and_save

 capture_youtube_live_frame_and_save (link:Optional[str]=None,
                                      place:Optional[str]=None)

Capture a jpeg file from YouTube Live and save in data directory

Type Default Details
link Optional None YouTube Live link
place Optional None Location of live image
Returns str Path to the saved image

source

ask_gpt4v_about_image_file

 ask_gpt4v_about_image_file (path:str)

Tell all about quantitative information from a given image file

Type Details
path str Path to the image file
Returns str JSON string with quantitative information

Utilities for GPT Function calling

We can use dynamic utilities functions to integrate this to GPT Function calling:

  • Parmater descriptions: extract parameter descriptions from a funcion
  • Parameter converter: convert Python parameter types into schema accepted formats
  • Schema generator: extract function information into tool schema to bet set for GPT
  • Function execution: execute function dynamically based on function names and input arguments

1. Extractor for parameter descriptions

source

extract_parameter_comments

 extract_parameter_comments (func:Callable)

Extract comments for function arguments

Type Details
func Callable Function to extract comments from
Returns dict Dictionary with parameter comments

Test usage with example functions:

print(f"Parameters of capture Youtube Live frame function: \
{extract_parameter_comments(capture_youtube_live_frame_and_save)}")
print(f"Parameters of ask GPT4V about image file function: \
{extract_parameter_comments(ask_gpt4v_about_image_file)}")
Parameters of capture Youtube Live frame function: {'link': 'YouTube Live link', 'place': 'Location of live image'}
Parameters of ask GPT4V about image file function: {'path': 'Path to the image file'}

2. Converter for Python parameter types to acceptable tool schema

source

param_converter

 param_converter (param_type, description)

Convert Python parameter types to acceptable types for tool schema

Type Details
param_type The type of the parameter
description The description of the parameter
Returns dict The converted parameter

Test usage with a more complicated data type Optional[str]:

param_schema = param_converter(Optional[str], "YouTube Live link")
print(json.dumps(param_schema, indent=2))
{
  "anyOf": [
    {
      "type": "string",
      "description": "YouTube Live link"
    },
    {
      "type": "null",
      "description": "A default value will be automatically used."
    }
  ]
}

3. Tool schema

source

tool_schema

 tool_schema (func:Callable, service_name:Optional[str]=None)

Automatically generate a schema from its parameters and docstring

Type Default Details
func Callable The function to generate the schema for
service_name Optional None The name of the service
Returns dict The generated tool schema 
[
  {
    "type": "function",
    "function": {
      "name": "capture_youtube_live_frame_and_save",
      "description": "Capture a jpeg file from YouTube Live and save in data directory",
      "parameters": {
        "type": "object",
        "properties": {
          "link": {
            "anyOf": [
              {
                "type": "string",
                "description": "YouTube Live link"
              },
              {
                "type": "null",
                "description": "A default value will be automatically used."
              }
            ]
          },
          "place": {
            "anyOf": [
              {
                "type": "string",
                "description": "Location of live image"
              },
              {
                "type": "null",
                "description": "A default value will be automatically used."
              }
            ]
          }
        },
        "required": []
      },
      "metadata": {
        "module": "__main__",
        "service": "__main__"
      }
    }
  },
  {
    "type": "function",
    "function": {
      "name": "ask_gpt4v_about_image_file",
      "description": "Tell all about quantitative information from a given image file",
      "parameters": {
        "type": "object",
        "properties": {
          "path": {
            "type": "string",
            "description": "Path to the image file"
          }
        },
        "required": [
          "path"
        ]
      },
      "metadata": {
        "module": "__main__",
        "service": "__main__"
      }
    }
  }
]

4. Excecution functions

source

fn_result_content

 fn_result_content (call, tools=[])

Create a content containing the result of the function call

source

fn_exec

 fn_exec (call, tools=[])

Execute the function call

source

fn_metadata

 fn_metadata (tool)

source

fn_args

 fn_args (call)

source

fn_name

 fn_name (call)

source

form_msgs

 form_msgs (msgs)

source

form_msg

 form_msg (role:Literal['system','user','assistant','tool'], content:str,
           tool_call_id:Optional[str]=None)

Create a message for the conversation

Type Default Details
role Literal The role of the message sender
content str The content of the message
tool_call_id Optional None The ID of the tool call (if role == “tool”)

source

complete

 complete (messages:list[dict], tools:list[dict]=[])

Complete the conversation with the given message

Type Default Details
messages list The list of messages
tools list [] The list of tools
Returns Tuple The role and content of the last message

Test with our existing toolbox:

# Test the function
messages = form_msgs([
    ("system", "You are a helpful system administrator. Use the supplied tools to assist the user."),
    ("user", "Hi, can you capture YouTube Live?")
])
complete(messages, YTLiveTools)
print_msgs(messages)
>> System:
You are a helpful system administrator. Use the supplied tools to assist the user.
>> User:
Hi, can you capture YouTube Live?
>> Assistant:
Yes, I can help capture a frame from a YouTube Live stream. Please provide the link to the YouTube
Live stream you want me to capture from.

Let’s try to continue this conversation:

messages.append(form_msg("user", "You can use the default link."))
complete(messages, YTLiveTools)
print_msgs(messages)
[youtube] Extracting URL: https://www.youtube.com/watch?v=LMZQ7eFhm58
[youtube] LMZQ7eFhm58: Downloading webpage
[youtube] LMZQ7eFhm58: Downloading ios player API JSON
[youtube] LMZQ7eFhm58: Downloading mweb player API JSON
[youtube] LMZQ7eFhm58: Downloading m3u8 information
[youtube] LMZQ7eFhm58: Downloading m3u8 information
cap_2024.11.18_22:33:39_unclear.jpg
>> System:
You are a helpful system administrator. Use the supplied tools to assist the user.
>> User:
Hi, can you capture YouTube Live?
>> Assistant:
Yes, I can help capture a frame from a YouTube Live stream. Please provide the link to the YouTube
Live stream you want me to capture from.
>> User:
You can use the default link.
>> Assistant:
I have captured a frame from the YouTube Live stream using the default link. If you need more
information or have any specific requests regarding the image, please let me know!
messages.append(form_msg("user", "Can you extract information from this file?"))
complete(messages, YTLiveTools)
print_msgs(messages)
>> System:
You are a helpful system administrator. Use the supplied tools to assist the user.
>> User:
Hi, can you capture YouTube Live?
>> Assistant:
Yes, I can help capture a frame from a YouTube Live stream. Please provide the link to the YouTube
Live stream you want me to capture from.
>> User:
You can use the default link.
>> Assistant:
I have captured a frame from the YouTube Live stream using the default link. If you need more
information or have any specific requests regarding the image, please let me know!
>> User:
Can you extract information from this file?
>> Assistant:
Here's the information extracted from the captured frame of the YouTube Live stream:  -
**Timestamp**: 2024-11-18T22:25:55 - **Location**: Tuomiokirkko - **Image Dimensions**: 1280x720
pixels  ### Scene Details: - **Buildings**:   - Number of buildings: 15   - Building height range:
3-5 stories - **Vehicles**:   - Number of vehicles: 0   - Number of available parking spaces: 0 -
**Water Bodies**: Not visible - **Street Lights**:   - Number of street lights: 10 - **People**:   -
Approximate number: 0 - **Lighting**:   - Time of day: Night   - Artificial lighting: Prominent -
**Visibility**: Clear - **Sky**:   - Visible: Yes   - Light conditions: Dark  If you need more
details or have other questions, feel free to ask!