Tool Manager Reference

The tool manager is the runtime that makes multiple nomad.fm_base_tool.TorchModuleTool models practical to serve from a single Nomad process. It keeps track of available devices, loads tools onto an accelerator when they need to run, and can offload idle tools back to CPU so GPU memory can be shared across a larger set of models.

It also queues requests, batches calls for tools that support batching, and can register the managed tools with FastMCP.

For managed Torch tools, the manager keeps one resident CPU instance when the tool is loaded from configuration and may create additional tool instances from the configured source when a busy tool is assigned to more than one device slot. Idle tools scale down one device allocation per idle_seconds interval. After a tool is fully removed from device slots, disk_idle_seconds controls when the resident CPU instance may be dropped; a later request reloads it with TorchModuleTool.from_pretrained(source). Separately, gc_idle_seconds controls when unused Python and accelerator caches are cleared while the server is idle.

nomad.torch_tool_manager.empty_accelerator_cache()

Release unused Python and accelerator memory back to the runtime.

Return type:

None

class nomad.torch_tool_manager.AcceleratorInfo(device, kind, index, name)

Metadata describing an available accelerator device.

Parameters:
device: device
kind: str
index: int | None
name: str | None
class nomad.torch_tool_manager.DeviceSlot(device, current_tool=None, tool=None, busy=False, last_used=<factory>)

Tracks which tool currently occupies a device.

Parameters:
device: device
current_tool: str | None
tool: TorchModuleTool | None
busy: bool
last_used: float
class nomad.torch_tool_manager.ToolRequest(input, future, enqueued_at=<factory>, metrics_recorded=False)

Represents a pending tool invocation.

Parameters:
input: BaseModel
future: Future
enqueued_at: float
metrics_recorded: bool
class nomad.torch_tool_manager.ToolState(name, registered_name, tool, cls, source, args_schema, output_schema, description, queue=<factory>, enqueued=False, batch_size=1, assigned_slots=<factory>, loaded_slots=<factory>, last_used=<factory>, resident_slot=None)

Queue and metadata for a registered tool.

Parameters:
name: str
registered_name: str
tool: TorchModuleTool | None
cls: type[TorchModuleTool]
source: str | Path
args_schema: type[BaseModel]
output_schema: type[BaseModel]
description: str
queue: deque[ToolRequest]
enqueued: bool
batch_size: int
assigned_slots: set[int]
loaded_slots: set[int]
last_used: float
resident_slot: int | None
reserve_slot(device_index)
Parameters:

device_index (int)

Return type:

None

release_slot(device_index)
Parameters:

device_index (int)

Return type:

None

mark_slot_loaded(device_index, *, now, is_resident)
Parameters:
Return type:

None

mark_slot_free(device_index, *, now, remains_loaded)
Parameters:
Return type:

None

load_new_tool(*, is_resident_load)
Parameters:

is_resident_load (bool)

Return type:

TorchModuleTool

class nomad.torch_tool_manager.TorchModelToolManager(config=None, *, device_provider=None)

Coordinate execution of multiple TorchModuleTool instances across devices.

Parameters:
property devices: list[device]

Devices managed by this instance.

property accelerator_info: list[AcceleratorInfo]

Detailed accelerator metadata.

register_tool(name, tool, *, source)

Register a tool for managed execution.

Parameters:
Return type:

None

add_to_fastmcp(server)

Register all managed tools with a FastMCP server.

Parameters:

server (FastMCP)

Return type:

dict[str, FunctionTool]

async call_tool(name, input=None, /, **kwargs)

Queue a call to name and await its result.

Parameters:
Return type:

BaseModel

async aclose()

Cancel background tasks and drain queues.

Return type:

None

queue_length_observations()
inflight_observations()
configured_batch_size_observations()
pending_tools_observations()
device_busy_observations()
loaded_tool_observations()
resident_tool_observations()
device_memory_allocated_observations()
device_memory_reserved_observations()
device_utilization_observations()