import glob
from llmcam.ytlive import capture_youtube_live_frame
from llmcam.gpt4v import ask_gpt4vFn to FC
python module to convert a given Fn into FC automatically
Example functions
For our first MVP, response generation mostly concern with GPT models answering generic questions and using a single tool for capturing and extracting information from a Youtube Livestream.
capture_youtube_live_frame_and_save
capture_youtube_live_frame_and_save (link:Optional[str]=None)
Capture a jpeg file from YouTube Live and save in data directory
| Type | Default | Details | |
|---|---|---|---|
| link | Optional | None | YouTube Live link |
| Returns | str | Path to the saved image |
ask_gpt4v_about_image_file
ask_gpt4v_about_image_file (path:str)
Tell all about quantitative information from a given image file
| Type | Details | |
|---|---|---|
| path | str | Path to the image file |
| Returns | str | JSON string with quantitative information |
Utilities for GPT Function calling
We can use dynamic utilities functions to integrate this to GPT Function calling:
- Parmater descriptions: extract parameter descriptions from a funcion
- Parameter converter: convert Python parameter types into schema accepted formats
- Schema generator: extract function information into tool schema to bet set for GPT
- Function execution: execute function dynamically based on function names and input arguments
1. Extractor for parameter descriptions
extract_parameter_comments
extract_parameter_comments (func:Callable)
Extract comments for function arguments
| Type | Details | |
|---|---|---|
| func | Callable | Function to extract comments from |
| Returns | dict | Dictionary with parameter comments |
Test usage with example functions:
print(f"Parameters of capture Youtube Live frame function: \
{extract_parameter_comments(capture_youtube_live_frame_and_save)}")
print(f"Parameters of ask GPT4V about image file function: \
{extract_parameter_comments(ask_gpt4v_about_image_file)}")Parameters of capture Youtube Live frame function: {'link': 'YouTube Live link'}
Parameters of ask GPT4V about image file function: {'path': 'Path to the image file'}2. Converter for Python parameter types to acceptable tool schema
param_converter
param_converter (param_type, description)
Convert Python parameter types to acceptable types for tool schema
| Type | Details | |
|---|---|---|
| param_type | The type of the parameter | |
| description | The description of the parameter | |
| Returns | dict | The converted parameter |
Test usage with a more complicated data type Optional[str]:
param_schema = param_converter(Optional[str], "YouTube Live link")
print(json.dumps(param_schema, indent=2)){
"anyOf": [
{
"type": "string",
"description": "YouTube Live link"
},
{
"type": "null",
"description": "A default value will be automatically used."
}
]
}3. Tool schema
tool_schema
tool_schema (func:Callable)
Automatically generate a schema from its parameters and docstring
| Type | Details | |
|---|---|---|
| func | Callable | The function to generate the schema for |
| Returns | dict | The generated tool schema |
[
{
"type": "function",
"function": {
"name": "capture_youtube_live_frame_and_save",
"description": "Capture a jpeg file from YouTube Live and save in data directory",
"parameters": {
"type": "object",
"properties": {
"link": {
"anyOf": [
{
"type": "string",
"description": "YouTube Live link"
},
{
"type": "null",
"description": "A default value will be automatically used."
}
]
}
},
"required": []
}
}
},
{
"type": "function",
"function": {
"name": "ask_gpt4v_about_image_file",
"description": "Tell all about quantitative information from a given image file",
"parameters": {
"type": "object",
"properties": {
"path": {
"type": "string",
"description": "Path to the image file"
}
},
"required": [
"path"
]
}
}
}
]4. Excecution functions
fn_result_content
fn_result_content (res)
Create a content containing the result of the function call
fn_exec
fn_exec (res)
fn_args
fn_args (res)
fn_name
fn_name (res)
generate_messages
generate_messages (message:str, history:list[dict]=[])
Generate messages from the user and the system
| Type | Default | Details | |
|---|---|---|---|
| message | str | New message frorm the user | |
| history | list | [] | Previous messages |
| Returns | list | List of messages |
Test usage with a conversation:
messages = generate_messages("Hi, can you capture YouTube Live and retrieve information from it? Use the default Youtube link.")
for message in messages:
print(f"{message['role'].capitalize()}: {message['content']}")[youtube] Extracting URL: https://www.youtube.com/watch?v=LMZQ7eFhm58
[youtube] LMZQ7eFhm58: Downloading webpage
[youtube] LMZQ7eFhm58: Downloading ios player API JSON
[youtube] LMZQ7eFhm58: Downloading mweb player API JSON
[youtube] LMZQ7eFhm58: Downloading m3u8 information
[youtube] LMZQ7eFhm58: Downloading m3u8 information
System: You are a helpful system administrator. Use the supplied tools to assist the user.
User: Hi, can you capture YouTube Live and retrieve information from it? Use the default Youtube link.
Assistant: None
Tool: {"link": null, "capture_youtube_live_frame_and_save": "../data/fail_2024.11.04_18:00:07_nowhere.jpg"}
Assistant: None
Tool: {"path": "../data/fail_2024.11.04_18:00:07_nowhere.jpg", "ask_gpt4v_about_image_file": "{\"timestamp\": \"2024-11-04T17:52:45\", \"dimensions\": {\"width\": 1280, \"height\": 720}, \"buildings\": {\"number_of_buildings\": 0}, \"vehicles\": {\"number_of_vehicles\": 0}, \"waterbodies\": {\"visible\": false}, \"street_lights\": {\"number_of_street_lights\": 5}, \"people\": {\"approximate_number\": 0}, \"lighting\": {\"time_of_day\": \"night\", \"artificial_lighting\": \"prominent\"}, \"visibility\": {\"clear\": false}, \"sky\": {\"visible\": true, \"light_conditions\": \"dark\"}}"}
Assistant: I captured a frame from the YouTube Live stream using the default link and retrieved the following information from the image:
- **Timestamp:** 2024-11-04T17:52:45
- **Image Dimensions:** 1280 x 720
- **Buildings:** There are no buildings visible in the image.
- **Vehicles:** There are no vehicles visible in the image.
- **Waterbodies:** No waterbodies are visible.
- **Street Lights:** There are 5 street lights visible.
- **People:** There are no people visible in the image.
- **Lighting:** It's night time with prominent artificial lighting.
- **Visibility:** The visibility is not clear.
- **Sky:** The sky is visible but it's dark.
If you have any other questions or need further assistance, feel free to ask!Let’s try to continue this conversation:
new_messages = generate_messages("Can you explain this timestamp", messages)
for message in new_messages:
print(f"{message['role'].capitalize()}: {message['content']}")System: You are a helpful system administrator. Use the supplied tools to assist the user.
User: Hi, can you capture YouTube Live and retrieve information from it? Use the default Youtube link.
Assistant: None
Tool: {"link": null, "capture_youtube_live_frame_and_save": "../data/fail_2024.11.04_18:00:07_nowhere.jpg"}
Assistant: None
Tool: {"path": "../data/fail_2024.11.04_18:00:07_nowhere.jpg", "ask_gpt4v_about_image_file": "{\"timestamp\": \"2024-11-04T17:52:45\", \"dimensions\": {\"width\": 1280, \"height\": 720}, \"buildings\": {\"number_of_buildings\": 0}, \"vehicles\": {\"number_of_vehicles\": 0}, \"waterbodies\": {\"visible\": false}, \"street_lights\": {\"number_of_street_lights\": 5}, \"people\": {\"approximate_number\": 0}, \"lighting\": {\"time_of_day\": \"night\", \"artificial_lighting\": \"prominent\"}, \"visibility\": {\"clear\": false}, \"sky\": {\"visible\": true, \"light_conditions\": \"dark\"}}"}
Assistant: I captured a frame from the YouTube Live stream using the default link and retrieved the following information from the image:
- **Timestamp:** 2024-11-04T17:52:45
- **Image Dimensions:** 1280 x 720
- **Buildings:** There are no buildings visible in the image.
- **Vehicles:** There are no vehicles visible in the image.
- **Waterbodies:** No waterbodies are visible.
- **Street Lights:** There are 5 street lights visible.
- **People:** There are no people visible in the image.
- **Lighting:** It's night time with prominent artificial lighting.
- **Visibility:** The visibility is not clear.
- **Sky:** The sky is visible but it's dark.
If you have any other questions or need further assistance, feel free to ask!
User: Can you explain this timestamp
Assistant: The timestamp "2024-11-04T17:52:45" follows the ISO 8601 format, which is a standard for representing date and time. Here's the breakdown:
- **2024-11-04:** This part represents the date. It means the 4th of November, 2024.
- **T:** The "T" is a separator indicating the end of the date and the start of the time component.
- **17:52:45:** This denotes the time. It is in a 24-hour clock format representing 5:52:45 PM.
This timestamp suggests the exact date and time when the image frame was captured from the YouTube Live stream. If you need more information or another type of analysis, feel free to ask!