snapshot of https://github.com/tldraw/tldraw.git (main, 2026-09-23; blobs > 131072B omitted)
Some checks failed
Deploy analytics worker / Deploy analytics worker to ${{ (github.ref == 'refs/heads/production' && 'production') || (github.ref == 'refs/heads/main' && 'staging') || 'preview' }} (push) Failing after 1s
i18n upload strings / Upload latest strings (push) Failing after 1s
Deploy bemo / Deploy bemo to ${{ (github.ref == 'refs/heads/bemo-production' && 'production') || (github.ref == 'refs/heads/main' && 'staging') || 'preview' }} (push) Failing after 1s
Deploy grafana resources / ${{ github.event_name == 'push' && 'Deploy grafana resources' || 'Validate grafana resources (dry run)' }} (push) Failing after 0s
End to end tests (dotcom) / Check for relevant changes (push) Failing after 0s
End to end tests (examples) / Check for relevant changes (push) Failing after 0s
End to end tests (mermaid readability) / Check for relevant changes (push) Failing after 1s
End to end tests (perf) / Check for relevant changes (push) Failing after 0s
Publish editor extensions / Publish editor extensions (push) Failing after 1s
Publish SDK packages / Remove the publish-packages label (push) Has been skipped
Checks / Tests & checks (push) Has been cancelled
Checks / Build all projects (push) Has been cancelled
Deploy templates / Deploy templates (push) Has been cancelled
End to end tests (dotcom) / End to end tests (dotcom) (push) Has been cancelled
End to end tests (mermaid readability) / End to end tests (mermaid readability) (push) Has been cancelled
End to end tests (perf) / End to end tests (perf) (push) Has been cancelled
Deploy .com / Deploy dotcom to ${{ (github.ref == 'refs/heads/production' && 'production') || (github.ref == 'refs/heads/main' && 'staging') || 'preview' }} (push) Has been cancelled
End to end tests (examples) / End to end tests (examples) (push) Has been cancelled
Publish SDK packages / Publish SDK packages (push) Has been cancelled
Publish SDK packages / Tell user released version (push) Has been cancelled
Publish SDK packages / Publishes code templates to separate repositories (push) Has been cancelled
Publish SDK packages / Deploys the template demos with the new SDK version (push) Has been cancelled

This commit is contained in:
upstream-mirror
2026-09-23 16:12:52 +05:00
commit a774ac673a
4541 changed files with 770499 additions and 0 deletions

View File

@@ -0,0 +1,114 @@
<div alt style="text-align: center; transform: scale(.5);">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/tldraw/tldraw/main/assets/github-hero-dark.png" />
<img alt="tldraw" src="https://raw.githubusercontent.com/tldraw/tldraw/main/assets/github-hero-light.png" />
</picture>
</div>
This starter kit builds a visual node-graph editor for AI image generation pipelines on top of [tldraw](https://github.com/tldraw/tldraw). Users place nodes on an infinite canvas, wire up typed ports, and run the graph to generate images — similar to ComfyUI, but built with React and tldraw.
## Environment setup
Create a `.dev.vars` file in the root directory and add your Replicate API key:
```
REPLICATE_API_TOKEN=your_replicate_api_token_here
```
Without a token, generation requests will fail with an error.
## Local development
Install dependencies with `yarn` or `npm install`.
Run the development server with `yarn dev` or `npm run dev`.
Open `http://localhost:5173/` in your browser to see the app.
## How it works
This starter kit has 3 main concepts:
1. **Nodes**: Nodes are shapes that accept inputs and produce outputs. There are 16 node types ranging from inputs (Model, Prompt, Load Image) to processing (Generate, ControlNet, Upscale) to display (Preview).
2. **Connections**: A connection joins the output of one node to the input of another. Connections are drawn as bezier curves and are color-coded by data type.
3. **Ports**: Ports are typed endpoints on nodes. Each port has a data type (`image`, `text`, `model`, `number`, `latent`, or `any`) and only compatible ports can be connected.
There are lots of useful interactions the user can take involving these concepts:
- Dragging from a port lets the user create a new connection
- Dragging from a port to empty space opens an on-canvas node picker
- Clicking in the middle of a connection lets the user insert a new node
- Connections enforce type compatibility — you can't connect an image port to a model port
- Cycle detection prevents circular dependencies
The app automatically detects groups of connected nodes and draws region overlays around them with play/stop controls. Each node also has a "play from here" button. The execution engine traverses the graph as a DAG, resolving dependencies and running nodes in order.
## File structure
- **`src/App.tsx`:** The main entry-point. Renders the `<Tldraw />` component with custom shape utils, a sidebar, and a default starter pipeline (Model → Prompt → Generate → Preview).
- **`src/nodes`:** Everything related to nodes.
- **`src/nodes/NodeShapeUtil.tsx`:** The node shape util, which defines a custom tldraw shape for our nodes.
- **`src/nodes/nodeTypes.tsx`:** The registry of all node type definitions.
- **`src/nodes/nodePorts.tsx`:** Helpers for working with the ports on a node and the values flowing through them.
- **`src/nodes/types/`:** Each node type definition. Nodes define their ports, default props, body height, execution logic, and React component.
- **`src/connection`:** Everything related to connections.
- **`src/connection/ConnectionShapeUtil.tsx`:** The connection shape util, which renders bezier curves between ports.
- **`src/connection/ConnectionBindingUtil.tsx`:** The connection binding util. Each connection has two bindings — one to the start node, and one to the end node.
- **`src/connection/insertNodeWithinConnection.tsx`:** A helper for inserting a new node in the middle of a connection.
- **`src/connection/keepConnectionsAtBottom.tsx`:** Side effect listeners that keep connections below nodes in the z-order.
- **`src/ports`:** Everything related to ports.
- **`src/ports/Port.tsx`:** Defines port types and a React component for rendering them.
- **`src/ports/PointingPort.tsx`:** An entry in tldraw's interaction state tree which extends the select tool with custom port interaction behavior.
- **`src/ports/portCompatibility.ts`:** Type compatibility rules for connecting ports.
- **`src/execution`:** Everything related to running workflows.
- **`src/execution/ExecutionGraph.tsx`:** The DAG execution engine that runs connected node graphs.
- **`src/execution/executionState.ts`:** Execution state management — tracking which nodes are running, results, and errors.
- **`src/components`:** UI and on-canvas React components.
- **`src/components/PipelineToolbar.tsx`:** A replacement for tldraw's default toolbar, rendered vertically on the left with draggable node types.
- **`src/components/ImagePipelineSidebar.tsx`:** A sidebar for saved templates and settings.
- **`src/components/OnCanvasNodePicker.tsx`:** A panel that appears on the canvas when dragging a connection to empty space, letting the user choose a compatible node to insert.
- **`src/components/PipelineRegions.tsx`:** Finds groups of connected nodes, draws bounding boxes around them, and provides play/stop controls.
- **`src/templates`:** Save and restore reusable subgraphs.
- **`worker/`:** Cloudflare Worker backend.
- **`worker/worker.ts`:** The main worker entry-point with API routes.
- **`worker/routes/`:** Route handlers for generate, upscale, style transfer, IP adapter, text generation, and image storage.
- **`worker/providers/`:** Provider abstractions for Replicate API calls.
## Deployment
This template uses a [Cloudflare Worker](https://developers.cloudflare.com/workers/) backend with [R2](https://developers.cloudflare.com/r2/) for image storage.
To deploy, create an R2 bucket and deploy the worker:
```bash
npx wrangler r2 bucket create image-pipeline
npx wrangler secret put REPLICATE_API_TOKEN
npm run build
npx wrangler deploy
```
## License
This project is part of the tldraw SDK. It is provided under the [tldraw SDK license](https://github.com/tldraw/tldraw/blob/main/LICENSE.md).
You can use the tldraw SDK in commercial or non-commercial projects so long as you preserve the "Made with tldraw" watermark on the canvas. To remove the watermark, you can purchase a [business license](https://tldraw.dev/pricing). Visit [tldraw.dev](https://tldraw.dev) to learn more.
## Trademarks
Copyright (c) 2025-present tldraw Inc. The tldraw name and logo are trademarks of tldraw. Please see our [trademark guidelines](https://github.com/tldraw/tldraw/blob/main/TRADEMARKS.md) for info on acceptable usage.
## Distributions
You can find tldraw on npm [here](https://www.npmjs.com/package/@tldraw/tldraw?activeTab=versions).
## Contribution
Found a bug? Please [submit an issue](https://github.com/tldraw/tldraw/issues/new).
## Community
Have questions, comments or feedback? [Join our discord](https://discord.gg/rhsyWMUJxd). For the latest news and release notes, visit [tldraw.dev](https://tldraw.dev).
## Contact
Find us on Twitter/X at [@tldraw](https://twitter.com/tldraw) or email us at [mailto:hello@tldraw.com](hello@tldraw.com).

View File

@@ -0,0 +1,13 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<link rel="icon" href="/favicon.ico" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>tldraw image pipeline</title>
</head>
<body>
<div id="root"></div>
<script type="module" src="/src/main.tsx"></script>
</body>
</html>

View File

@@ -0,0 +1,51 @@
{
"name": "tldraw-image-pipeline",
"private": true,
"version": "0.0.0",
"author": {
"name": "tldraw GB Ltd.",
"email": "hello@tldraw.com"
},
"homepage": "https://tldraw.dev",
"license": "MIT",
"type": "module",
"tldraw_template": {
"repo": "tldraw/image-pipeline-template",
"cli": {
"name": "Image pipeline",
"description": "Create a node-based builder for image generation pipelines.",
"shortDescription": "Visual node-based builder for image pipelines.",
"order": 6
},
"deploy": {
"target": "cloudflare",
"host": "image-pipeline.templates.tldraw.dev"
},
"scripts": {
"lint": null
}
},
"scripts": {
"dev": "vite --host",
"build": "tsc && vite build",
"lint": "yarn run -T tsx ../../internal/scripts/lint.ts",
"preview": "vite preview"
},
"dependencies": {
"classnames": "^2.5.1",
"itty-router": "^5.0.18",
"radix-ui": "^1.4.2",
"react": "^19.2.1",
"react-dom": "^19.2.1",
"tldraw": "workspace:*"
},
"devDependencies": {
"@cloudflare/vite-plugin": "^1.29.1",
"@types/react": "^19.2.7",
"@types/react-dom": "^19.2.3",
"@vitejs/plugin-react": "^6.0.1",
"typescript": "^5.8.3",
"vite": "^8.0.16",
"wrangler": "^4.75.0"
}
}

View File

@@ -0,0 +1,136 @@
import { useState } from 'react'
import { createShapeId, Editor, TLComponents, Tldraw, TldrawOptions } from 'tldraw'
import { ImagePipelineSidebar } from './components/ImagePipelineSidebar.tsx'
import { OnCanvasNodePicker } from './components/OnCanvasNodePicker.tsx'
import { PipelineRegions } from './components/PipelineRegions.tsx'
import { overrides, PipelineToolbar } from './components/PipelineToolbar.tsx'
import { ConnectionBindingUtil } from './connection/ConnectionBindingUtil'
import { ConnectionCenterHandleOverlayUtil } from './connection/ConnectionCenterHandleOverlayUtil'
import { ConnectionShapeUtil } from './connection/ConnectionShapeUtil'
import { keepConnectionsAtBottom } from './connection/keepConnectionsAtBottom'
import { disableTransparency } from './disableTransparency.tsx'
import { NodeShapeUtil } from './nodes/NodeShapeUtil'
import { PointingPort } from './ports/PointingPort'
const shapeUtils = [NodeShapeUtil, ConnectionShapeUtil]
const bindingUtils = [ConnectionBindingUtil]
const overlayUtils = [ConnectionCenterHandleOverlayUtil]
const components: TLComponents = {
InFrontOfTheCanvas: () => (
<>
<OnCanvasNodePicker />
<PipelineRegions />
</>
),
Toolbar: PipelineToolbar,
}
const options: Partial<TldrawOptions> = {
actionShortcutsLocation: 'menu',
maxPages: 1,
}
function App() {
const [editor, setEditor] = useState<Editor | null>(null)
return (
<div className="image-pipeline-layout" style={{ position: 'fixed', inset: 0 }}>
<div className="image-pipeline-sidebar">
{editor ? <ImagePipelineSidebar editor={editor} /> : <div />}
</div>
<div className="image-pipeline-canvas">
<Tldraw
persistenceKey="image-pipeline"
options={options}
overrides={overrides}
shapeUtils={shapeUtils}
bindingUtils={bindingUtils}
overlayUtils={overlayUtils}
components={components}
onMount={(editor) => {
;(window as any).editor = editor
setEditor(editor)
// Create a default pipeline if the canvas is empty
if (!editor.getCurrentPageShapes().some((s) => s.type === 'node')) {
createDefaultPipeline(editor)
}
editor.user.updateUserPreferences({ isSnapMode: true })
editor.getStateDescendant('select')!.addChild(PointingPort)
keepConnectionsAtBottom(editor)
disableTransparency(editor, ['connection'])
}}
/>
</div>
</div>
)
}
/**
* Create a default text-to-image pipeline to get users started.
*/
function createDefaultPipeline(editor: Editor) {
const modelId = createShapeId()
const promptId = createShapeId()
const generateId = createShapeId()
const previewId = createShapeId()
editor.createShapes([
{
id: modelId,
type: 'node',
x: 100,
y: 200,
props: {
node: { type: 'model', provider: 'flux', modelId: 'flux-dev' },
},
},
{
id: promptId,
type: 'node',
x: 100,
y: 450,
props: {
node: {
type: 'prompt',
text: 'a photo of a cat sitting on a windowsill',
},
},
},
{
id: generateId,
type: 'node',
x: 450,
y: 200,
props: {
node: {
type: 'generate',
steps: 20,
cfgScale: 7,
seed: Math.floor(Math.random() * 99999),
lastResultUrl: null,
},
},
},
{
id: previewId,
type: 'node',
x: 800,
y: 200,
props: {
node: {
type: 'preview',
lastImageUrl: null,
},
},
},
])
}
export default App

View File

@@ -0,0 +1,181 @@
/**
* Frontend API client for calling the Cloudflare Worker backend.
* Each function corresponds to a worker endpoint.
*/
export interface GenerateParams {
model: string
prompt: string
negativePrompt?: string
steps?: number
cfgScale?: number
seed?: number
controlNetMode?: string
controlNetStrength?: number
referenceImageUrl?: string
}
export interface GenerateResult {
imageUrl: string
seed: number
}
/**
* Call the /api/generate endpoint to create an AI-generated image.
* Falls back to a local placeholder if the worker is not available.
*/
export async function apiGenerate(params: GenerateParams): Promise<GenerateResult> {
try {
const response = await fetch('/api/generate', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(params),
})
if (!response.ok) {
const err = await response.json().catch(() => ({ error: response.statusText }))
throw new Error((err as { error?: string }).error ?? 'Generation failed')
}
return (await response.json()) as GenerateResult
} catch (e) {
throw new Error(`Backend unavailable: ${e instanceof Error ? e.message : e}`)
}
}
export interface UpscaleParams {
imageUrl: string
scale: number
method: string
}
export interface UpscaleResult {
imageUrl: string
}
/**
* Call the /api/upscale endpoint to upscale an image.
*/
export async function apiUpscale(params: UpscaleParams): Promise<UpscaleResult> {
try {
const response = await fetch('/api/upscale', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(params),
})
if (!response.ok) {
const err = await response.json().catch(() => ({ error: response.statusText }))
throw new Error((err as { error?: string }).error ?? 'Upscale failed')
}
return (await response.json()) as UpscaleResult
} catch (e) {
throw new Error(`Backend unavailable: ${e instanceof Error ? e.message : e}`)
}
}
export interface IPAdapterParams {
imageUrl: string
prompt: string
scale: number
steps: number
}
export interface IPAdapterResult {
imageUrl: string
}
/**
* Call the /api/ip-adapter endpoint to generate an image guided by a reference.
*/
export async function apiIPAdapter(params: IPAdapterParams): Promise<IPAdapterResult> {
try {
const response = await fetch('/api/ip-adapter', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(params),
})
if (!response.ok) {
const err = await response.json().catch(() => ({ error: response.statusText }))
throw new Error((err as { error?: string }).error ?? 'IP-Adapter failed')
}
return (await response.json()) as IPAdapterResult
} catch (e) {
throw new Error(`Backend unavailable: ${e instanceof Error ? e.message : e}`)
}
}
export interface StyleTransferParams {
styleImageUrl: string
contentImageUrl?: string
prompt?: string
model: string
strength: number
}
export interface StyleTransferResult {
imageUrl: string
}
/**
* Call the /api/style-transfer endpoint to transfer style between images.
*/
export async function apiStyleTransfer(params: StyleTransferParams): Promise<StyleTransferResult> {
try {
const response = await fetch('/api/style-transfer', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(params),
})
if (!response.ok) {
const err = await response.json().catch(() => ({ error: response.statusText }))
throw new Error((err as { error?: string }).error ?? 'Style transfer failed')
}
return (await response.json()) as StyleTransferResult
} catch (e) {
throw new Error(`Backend unavailable: ${e instanceof Error ? e.message : e}`)
}
}
export interface GenerateTextParams {
input?: string
prompt: string
}
export interface GenerateTextResult {
text: string
}
/**
* Call the /api/generate-text endpoint to generate text from a multimodal AI model.
* Falls back to a local placeholder if the worker is not available.
*/
export async function apiGenerateText(params: GenerateTextParams): Promise<GenerateTextResult> {
// Coerce input to string so the worker always receives a string
const coercedParams = {
...params,
input: params.input != null ? String(params.input) : undefined,
}
try {
const response = await fetch('/api/generate-text', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(coercedParams),
})
if (!response.ok) {
const err = await response.json().catch(() => ({ error: response.statusText }))
throw new Error((err as { error?: string }).error ?? 'Text generation failed')
}
return (await response.json()) as GenerateTextResult
} catch (e) {
throw new Error(`Backend unavailable: ${e instanceof Error ? e.message : e}`)
}
}

View File

@@ -0,0 +1,174 @@
import { useRef } from 'react'
import { createShapeId, Editor, getPointerInfo, onDragFromToolbarToCreateShape, Vec } from 'tldraw'
import { getNodeDefinitions, NodeType } from '../nodes/nodeTypes'
const CATEGORY_LABELS: Record<string, string> = {
input: 'Input',
process: 'Process',
output: 'Output',
utility: 'Utility',
}
const CATEGORY_ORDER = ['input', 'process', 'output', 'utility']
const DRAG_DISTANCE_SQ = 36 // 6px
function createNodeAtCenter(editor: Editor, node: NodeType) {
const shapeId = createShapeId()
editor.run(() => {
editor.createShape({
id: shapeId,
type: 'node',
props: { node },
})
const shapeBounds = editor.getShapePageBounds(shapeId)!
const center = editor.getViewportPageBounds().center
editor.updateShape({
id: shapeId,
type: 'node',
x: center.x - shapeBounds.width / 2,
y: center.y - shapeBounds.height / 2,
})
editor.select(shapeId)
})
}
function SidebarItem({
editor,
title,
icon,
getDefault,
}: {
editor: Editor
title: string
icon: React.ReactElement
getDefault: () => NodeType
}) {
const stateRef = useRef<
| { name: 'idle' }
| { name: 'pointing'; start: { x: number; y: number } }
| { name: 'dragging' }
| { name: 'dragged' }
>({ name: 'idle' })
return (
<button
className="ImagePipelineSidebar-item"
onPointerDown={(e) => {
stateRef.current = {
name: 'pointing',
start: { x: e.clientX, y: e.clientY },
}
e.currentTarget.setPointerCapture(e.pointerId)
}}
onPointerMove={(e) => {
if (stateRef.current.name === 'pointing') {
const dist = Vec.Dist2(stateRef.current.start, {
x: e.clientX,
y: e.clientY,
})
if (dist > DRAG_DISTANCE_SQ) {
const start = stateRef.current.start
stateRef.current = { name: 'dragging' }
editor.run(() => {
editor.setCurrentTool('select')
editor.dispatch({
type: 'pointer',
target: 'canvas',
name: 'pointer_down',
...getPointerInfo(editor, e.nativeEvent),
point: start,
})
editor.selectNone()
onDragFromToolbarToCreateShape(
editor,
{
type: 'pointer',
target: 'canvas',
name: 'pointer_move',
...getPointerInfo(editor, e.nativeEvent),
point: start,
},
{
createShape: (id) => {
editor.createShape({
id,
type: 'node',
props: { node: getDefault() },
})
},
}
)
editor.getContainer().focus()
})
}
}
}}
onPointerUp={(e) => {
e.currentTarget.releasePointerCapture(e.pointerId)
if (stateRef.current.name === 'dragging') {
editor.dispatch({
type: 'pointer',
target: 'canvas',
name: 'pointer_up',
...getPointerInfo(editor, e.nativeEvent),
})
stateRef.current = { name: 'dragged' }
return
}
stateRef.current = { name: 'idle' }
}}
onClick={() => {
if (stateRef.current.name === 'dragged') {
stateRef.current = { name: 'idle' }
return
}
stateRef.current = { name: 'idle' }
createNodeAtCenter(editor, getDefault())
}}
>
<span className="ImagePipelineSidebar-item-icon">{icon}</span>
<span className="ImagePipelineSidebar-item-title">{title}</span>
</button>
)
}
export function ImagePipelineSidebar({ editor }: { editor: Editor }) {
const defs = getNodeDefinitions(editor)
// Group definitions by category
const grouped: Record<string, (typeof defs)[keyof typeof defs][]> = {}
for (const def of Object.values(defs)) {
const cat = def.category
if (!grouped[cat]) grouped[cat] = []
if (def.hidden) continue
grouped[cat].push(def)
}
return (
<div className="ImagePipelineSidebar tl-theme__light">
<div className="ImagePipelineSidebar-header">Nodes</div>
<div className="ImagePipelineSidebar-list">
{CATEGORY_ORDER.map((cat) => {
const items = grouped[cat]
if (!items?.length) return null
return (
<div key={cat} className="ImagePipelineSidebar-group">
<div className="ImagePipelineSidebar-category">{CATEGORY_LABELS[cat] ?? cat}</div>
{items.map((def) => (
<SidebarItem
key={def.type}
editor={editor}
title={def.title}
icon={def.icon}
getDefault={() => def.getDefault()}
/>
))}
</div>
)
})}
</div>
</div>
)
}

View File

@@ -0,0 +1,181 @@
import { Dialog, VisuallyHidden } from 'radix-ui'
import { useCallback, useMemo, useState } from 'react'
import {
TldrawUiButton,
TldrawUiButtonIcon,
TldrawUiButtonLabel,
TldrawUiMenuContextProvider,
TldrawUiMenuGroup,
TLShapeId,
useEditor,
usePassThroughWheelEvents,
useQuickReactor,
useValue,
Vec,
VecModel,
} from 'tldraw'
import { getConnectionTerminals } from '../connection/ConnectionShapeUtil'
import { NODE_WIDTH_PX } from '../constants'
import { getNodeDefinitions, NodeType } from '../nodes/nodeTypes'
import { NodeDefinition } from '../nodes/types/shared'
import { EditorAtom } from '../utils'
export interface OnCanvasNodePickerState {
connectionShapeId: TLShapeId
location: 'start' | 'end' | 'middle'
onPick: (nodeType: NodeType, position: VecModel) => void
onClose: () => void
}
export const onCanvasNodePickerState = new EditorAtom<OnCanvasNodePickerState | null>(
'on canvas node picker',
() => null
)
export function OnCanvasNodePicker() {
const editor = useEditor()
const onClose = useCallback(() => {
const state = onCanvasNodePickerState.get(editor)
if (!state) return
onCanvasNodePickerState.set(editor, null)
state.onClose()
}, [editor])
const nodeDefs = getNodeDefinitions(editor)
return (
<OnCanvasNodePickerDialog onClose={onClose}>
<TldrawUiMenuGroup id="inputs">
<OnCanvasNodePickerItem definition={nodeDefs.model} onClose={onClose} />
<OnCanvasNodePickerItem definition={nodeDefs.prompt} onClose={onClose} />
<OnCanvasNodePickerItem definition={nodeDefs.load_image} onClose={onClose} />
</TldrawUiMenuGroup>
<TldrawUiMenuGroup id="process">
<OnCanvasNodePickerItem definition={nodeDefs.generate} onClose={onClose} />
<OnCanvasNodePickerItem definition={nodeDefs.controlnet} onClose={onClose} />
<OnCanvasNodePickerItem definition={nodeDefs.blend} onClose={onClose} />
<OnCanvasNodePickerItem definition={nodeDefs.adjust} onClose={onClose} />
<OnCanvasNodePickerItem definition={nodeDefs.prompt_concat} onClose={onClose} />
</TldrawUiMenuGroup>
<TldrawUiMenuGroup id="output">
<OnCanvasNodePickerItem definition={nodeDefs.preview} onClose={onClose} />
</TldrawUiMenuGroup>
</OnCanvasNodePickerDialog>
)
}
function OnCanvasNodePickerDialog({
children,
onClose,
}: {
children: React.ReactNode
onClose: () => void
}) {
const editor = useEditor()
const location = useValue('location', () => onCanvasNodePickerState.get(editor)?.location, [
editor,
])
const shouldRender = !!location
const [container, setContainer] = useState<HTMLDivElement | null>(null)
usePassThroughWheelEvents(useMemo(() => ({ current: container }), [container]))
useQuickReactor(
'OnCanvasNodePicker',
() => {
const state = onCanvasNodePickerState.get(editor)
if (!state) return
if (!container) return
const connection = editor.getShape(state.connectionShapeId)
if (!connection || !editor.isShapeOfType(connection, 'connection')) {
onClose()
return
}
const terminals = getConnectionTerminals(editor, connection)
const terminalInConnectionSpace =
state.location === 'middle'
? Vec.Lrp(terminals.start, terminals.end, 0.5)
: terminals[state.location]
const terminalInPageSpace = editor
.getShapePageTransform(connection)
.applyToPoint(terminalInConnectionSpace)
const terminalInViewportSpace = editor.pageToViewport(terminalInPageSpace)
container.style.transform = `translate(${terminalInViewportSpace.x}px, ${terminalInViewportSpace.y}px) scale(${editor.getZoomLevel()}) `
},
[editor, container]
)
return (
<Dialog.Root
open={shouldRender}
modal={false}
onOpenChange={(isOpen) => {
if (!isOpen) onClose()
}}
>
<Dialog.Content
ref={setContainer}
className={`OnCanvasNodePicker OnCanvasNodePicker_${location}`}
style={{ width: NODE_WIDTH_PX }}
>
<div className="OnCanvasNodePicker-content">
<VisuallyHidden.Root>
<Dialog.Title>Insert node</Dialog.Title>
</VisuallyHidden.Root>
<TldrawUiMenuContextProvider sourceId="dialog" type="menu">
{children}
</TldrawUiMenuContextProvider>
</div>
</Dialog.Content>
</Dialog.Root>
)
}
function OnCanvasNodePickerItem<T extends NodeType>({
definition,
onClose,
}: {
definition: NodeDefinition<T>
onClose: () => void
}) {
const editor = useEditor()
return (
<TldrawUiButton
key={definition.type}
type="menu"
className="OnCanvasNodePicker-button"
onPointerDown={editor.markEventAsHandled}
onClick={() => {
const state = onCanvasNodePickerState.get(editor)
if (!state) return
const connection = editor.getShape(state.connectionShapeId)
if (!connection || !editor.isShapeOfType(connection, 'connection')) {
onClose()
return
}
const terminals = getConnectionTerminals(editor, connection)
const terminalInConnectionSpace =
state.location === 'middle'
? Vec.Lrp(terminals.start, terminals.end, 0.5)
: terminals[state.location]
const terminalInPageSpace = editor
.getShapePageTransform(connection)
.applyToPoint(terminalInConnectionSpace)
state.onPick(definition.getDefault(), terminalInPageSpace)
onClose()
}}
>
<TldrawUiButtonIcon icon={definition.icon} />
<TldrawUiButtonLabel>{definition.title}</TldrawUiButtonLabel>
</TldrawUiButton>
)
}

View File

@@ -0,0 +1,150 @@
import classNames from 'classnames'
import { useRef } from 'react'
import {
Box,
BoxModel,
Editor,
TLShapeId,
useEditor,
usePassThroughWheelEvents,
useQuickReactor,
useValue,
} from 'tldraw'
import { executionState, startExecution, stopExecution } from '../execution/executionState'
import { getNodePortConnections } from '../nodes/nodePorts'
import { NodeShape } from '../nodes/NodeShapeUtil'
import { PlayIcon } from './icons/PlayIcon'
import { StopIcon } from './icons/StopIcon'
interface PipelineRegion {
bounds: BoxModel
nodes: Set<TLShapeId>
startingNodes: Set<TLShapeId>
}
function findPipelineRegions(editor: Editor): PipelineRegion[] {
const regionsByShape = new Map<TLShapeId, Set<TLShapeId>>()
const visitedNodes = new Set<TLShapeId>()
function visit(node: NodeShape, currentRegion?: Set<TLShapeId>) {
if (visitedNodes.has(node.id)) return
visitedNodes.add(node.id)
if (!currentRegion) {
currentRegion = new Set()
}
regionsByShape.set(node.id, currentRegion)
currentRegion.add(node.id)
for (const connection of getNodePortConnections(editor, node)) {
visit(editor.getShape(connection.connectedShapeId) as NodeShape, currentRegion)
}
}
for (const node of editor.getCurrentPageShapes()) {
if (editor.isShapeOfType(node, 'node')) {
visit(node)
}
}
return Array.from(new Set(regionsByShape.values()), (nodeIds): PipelineRegion => {
let bounds: Box | null = null
const startingNodes = new Set<TLShapeId>()
for (const nodeId of nodeIds) {
const hasInputs = getNodePortConnections(editor, nodeId).some((c) => c.terminal === 'end')
if (!hasInputs) {
startingNodes.add(nodeId)
}
const nodeBounds = editor.getShapePageBounds(nodeId)
if (!nodeBounds) continue
if (bounds) {
bounds.union(nodeBounds)
} else {
bounds = Box.From(nodeBounds)
}
}
return {
bounds: bounds!.expandBy(30),
nodes: nodeIds,
startingNodes,
}
}).filter((w) => w.nodes.size > 1)
}
export function PipelineRegions() {
const editor = useEditor()
const regions = useValue('regions', () => findPipelineRegions(editor), [editor])
return regions.map((region, i) => <PipelineRegionComponent key={i} region={region} />)
}
function PipelineRegionComponent({ region }: { region: PipelineRegion }) {
const editor = useEditor()
const ref = useRef<HTMLDivElement>(null)
const isExecuting = useValue(
'isExecuting',
() => {
const execution = executionState.get(editor).runningGraph
if (!execution) return false
for (const nodeId of region.nodes) {
if (execution.getNodeStatus(nodeId) === 'executing') {
return true
}
}
return false
},
[editor, region]
)
useQuickReactor(
'PipelineRegion positioning',
() => {
if (!ref.current) return
const camera = editor.getCamera()
if (camera.z < 0.25) {
ref.current.style.display = 'none'
return
} else {
ref.current.style.display = 'block'
}
const position = editor.pageToViewport(region.bounds)
ref.current.style.transform = `translate(${position.x}px, ${position.y}px)`
ref.current.style.width = `${region.bounds.w * camera.z}px`
ref.current.style.height = `${region.bounds.h * camera.z}px`
},
[region, editor]
)
usePassThroughWheelEvents(ref)
return (
<div
className={classNames('PipelineRegion', { PipelineRegion_executing: isExecuting })}
ref={ref}
>
<button
className="PipelineRegion-button"
onPointerDown={editor.markEventAsHandled}
onClick={() => {
if (isExecuting) {
stopExecution(editor)
} else {
startExecution(editor, region.startingNodes)
}
}}
>
{isExecuting ? <StopIcon /> : <PlayIcon />}
</button>
</div>
)
}

View File

@@ -0,0 +1,109 @@
import {
AssetToolbarItem,
createShapeId,
DefaultActionsMenu,
DefaultQuickActions,
DefaultToolbar,
DrawToolbarItem,
Editor,
HandToolbarItem,
NoteToolbarItem,
onDragFromToolbarToCreateShape,
RectangleToolbarItem,
SelectToolbarItem,
TextToolbarItem,
TldrawUiMenuGroup,
TLShapeId,
TLUiOverrides,
ToolbarItem,
useEditor,
Vec,
} from 'tldraw'
import { NodeShape } from '../nodes/NodeShapeUtil'
import { getNodeDefinitions, NodeType } from '../nodes/nodeTypes'
import { TemplatePicker } from './TemplatePicker'
function createNodeShape(editor: Editor, shapeId: TLShapeId, center: Vec, node: NodeType) {
const markId = editor.markHistoryStoppingPoint('create node')
editor.run(() => {
editor.createShape({
id: shapeId,
type: 'node',
props: { node },
})
const shape = editor.getShape<NodeShape>(shapeId)!
const shapeBounds = editor.getShapePageBounds(shapeId)!
const x = center.x - shapeBounds.width / 2
const y = center.y - shapeBounds.height / 2
editor.updateShape({ ...shape, x, y })
editor.select(shapeId)
})
return markId
}
export const overrides: TLUiOverrides = {
tools: (editor, tools, _) => {
for (const nodeDef of Object.values(getNodeDefinitions(editor))) {
if (nodeDef.hidden) continue
tools[`node-${nodeDef.type}`] = {
id: `node-${nodeDef.type}`,
label: nodeDef.title,
icon: nodeDef.icon,
onSelect: () => {
createNodeShape(
editor,
createShapeId(),
editor.getViewportPageBounds().center,
nodeDef.getDefault()
)
},
onDragStart: (_, info) => {
onDragFromToolbarToCreateShape(editor, info, {
createShape: (id) => {
editor.createShape({
id,
type: 'node',
props: { node: nodeDef.getDefault() },
})
},
})
},
}
}
return tools
},
}
export function PipelineToolbar() {
const editor = useEditor()
const nodeDefs = Object.values(getNodeDefinitions(editor)).filter((d) => !d.hidden)
return (
<DefaultToolbar>
<TldrawUiMenuGroup id="selection">
<SelectToolbarItem />
<HandToolbarItem />
</TldrawUiMenuGroup>
<TldrawUiMenuGroup id="shapes">
<DrawToolbarItem />
<NoteToolbarItem />
<RectangleToolbarItem />
<TextToolbarItem />
<AssetToolbarItem />
</TldrawUiMenuGroup>
<TldrawUiMenuGroup id="nodes">
{nodeDefs.map((nodeDef) => (
<ToolbarItem key={nodeDef.type} tool={`node-${nodeDef.type}`} />
))}
</TldrawUiMenuGroup>
<TemplatePicker />
<DefaultQuickActions />
<DefaultActionsMenu />
</DefaultToolbar>
)
}

View File

@@ -0,0 +1,113 @@
import { useCallback, useEffect, useState } from 'react'
import {
TldrawUiButton,
TldrawUiButtonIcon,
TldrawUiButtonLabel,
TldrawUiPopover,
TldrawUiPopoverContent,
TldrawUiPopoverTrigger,
useEditor,
} from 'tldraw'
import { saveSelectionAsTemplate, stampTemplate } from '../templates/templateActions'
import { deleteTemplate, loadTemplates, PipelineTemplate } from '../templates/templateState'
import { TemplateIcon } from './icons/TemplateIcon'
/**
* A popover-style template picker shown in the toolbar area.
*/
export function TemplatePicker() {
const editor = useEditor()
const [isOpen, setIsOpen] = useState(false)
const [templates, setTemplates] = useState<PipelineTemplate[]>([])
const [saveName, setSaveName] = useState('')
const refresh = useCallback(() => {
setTemplates(loadTemplates())
}, [])
useEffect(() => {
if (isOpen) refresh()
}, [isOpen, refresh])
const onSave = useCallback(() => {
if (!saveName.trim()) return
const result = saveSelectionAsTemplate(editor, saveName.trim(), '')
if (result) {
setSaveName('')
refresh()
}
}, [editor, saveName, refresh])
const onStamp = useCallback(
(template: PipelineTemplate) => {
const center = editor.getViewportPageBounds().center
stampTemplate(editor, template, center)
setIsOpen(false)
},
[editor]
)
const onDelete = useCallback(
(id: string) => {
deleteTemplate(id)
refresh()
},
[refresh]
)
return (
<TldrawUiPopover id="template-picker" open={isOpen} onOpenChange={setIsOpen}>
<TldrawUiPopoverTrigger>
<TldrawUiButton type="icon" title="Templates">
<TldrawUiButtonIcon icon={<TemplateIcon />} />
</TldrawUiButton>
</TldrawUiPopoverTrigger>
<TldrawUiPopoverContent side="bottom" align="start" sideOffset={8}>
<div className="TemplatePicker-popover">
<div className="TemplatePicker-header">Templates</div>
<div className="TemplatePicker-save">
<input
type="text"
placeholder="Name for selection..."
value={saveName}
onChange={(e) => setSaveName(e.target.value)}
onKeyDown={(e) => {
if (e.key === 'Enter') onSave()
}}
/>
<TldrawUiButton type="normal" onClick={onSave}>
<TldrawUiButtonLabel>Save</TldrawUiButtonLabel>
</TldrawUiButton>
</div>
<div className="TemplatePicker-list">
{templates.length === 0 ? (
<div className="TemplatePicker-empty">Select 2+ nodes and save as a template</div>
) : (
templates.map((t) => (
<div key={t.id} className="TemplatePicker-item">
<TldrawUiButton
type="menu"
className="TemplatePicker-item-button"
onClick={() => onStamp(t)}
>
<TldrawUiButtonIcon icon={<TemplateIcon />} />
<TldrawUiButtonLabel>
{t.name} ({t.nodes.length} nodes)
</TldrawUiButtonLabel>
</TldrawUiButton>
<TldrawUiButton
type="icon"
title="Delete template"
onClick={() => onDelete(t.id)}
>
<TldrawUiButtonLabel>×</TldrawUiButtonLabel>
</TldrawUiButton>
</div>
))
)}
</div>
</div>
</TldrawUiPopoverContent>
</TldrawUiPopover>
)
}

View File

@@ -0,0 +1,7 @@
export function AddIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M8 3V13M3 8H13" stroke="currentColor" strokeWidth="1.5" strokeLinecap="round" />
</svg>
)
}

View File

@@ -0,0 +1,12 @@
export function AdjustIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M2 4H6M10 4H14" stroke="currentColor" strokeWidth="1.33" strokeLinecap="round" />
<circle cx="8" cy="4" r="1.5" stroke="currentColor" strokeWidth="1.33" />
<path d="M2 8H4M8 8H14" stroke="currentColor" strokeWidth="1.33" strokeLinecap="round" />
<circle cx="6" cy="8" r="1.5" stroke="currentColor" strokeWidth="1.33" />
<path d="M2 12H9M13 12H14" stroke="currentColor" strokeWidth="1.33" strokeLinecap="round" />
<circle cx="11" cy="12" r="1.5" stroke="currentColor" strokeWidth="1.33" />
</svg>
)
}

View File

@@ -0,0 +1,8 @@
export function BlendIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<circle cx="6.5" cy="7" r="4" stroke="currentColor" strokeWidth="1.33" />
<circle cx="9.5" cy="9" r="4" stroke="currentColor" strokeWidth="1.33" />
</svg>
)
}

View File

@@ -0,0 +1,13 @@
export function CaptureIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<path
d="M2 5.5V3.5C2 2.67 2.67 2 3.5 2H5.5M10.5 2H12.5C13.33 2 14 2.67 14 3.5V5.5M14 10.5V12.5C14 13.33 13.33 14 12.5 14H10.5M5.5 14H3.5C2.67 14 2 13.33 2 12.5V10.5"
stroke="currentColor"
strokeWidth="1.33"
strokeLinecap="round"
strokeLinejoin="round"
/>
</svg>
)
}

View File

@@ -0,0 +1,16 @@
export function ControlNetIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<rect x="2" y="2" width="12" height="12" rx="2" stroke="currentColor" strokeWidth="1.2" />
<path
d="M5 11L7 7L9 9L11 5"
stroke="currentColor"
strokeWidth="1.2"
strokeLinecap="round"
strokeLinejoin="round"
/>
<circle cx="5" cy="11" r="1" fill="currentColor" />
<circle cx="11" cy="5" r="1" fill="currentColor" />
</svg>
)
}

View File

@@ -0,0 +1,14 @@
export function GenerateIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<path
d="M8 2V4M8 12V14M2 8H4M12 8H14M4.22 4.22L5.64 5.64M10.36 10.36L11.78 11.78M11.78 4.22L10.36 5.64M5.64 10.36L4.22 11.78"
stroke="currentColor"
strokeWidth="1.33"
strokeLinecap="round"
strokeLinejoin="round"
/>
<circle cx="8" cy="8" r="2" stroke="currentColor" strokeWidth="1.33" />
</svg>
)
}

View File

@@ -0,0 +1,17 @@
export function GenerateTextIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<path
d="M3 4H10M3 8H8M3 12H6"
stroke="currentColor"
strokeWidth="1.33"
strokeLinecap="round"
strokeLinejoin="round"
/>
<path
d="M12.5 7L13.1 8.4L14.5 9L13.1 9.6L12.5 11L11.9 9.6L10.5 9L11.9 8.4L12.5 7Z"
fill="currentColor"
/>
</svg>
)
}

View File

@@ -0,0 +1,16 @@
export function IPAdapterIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<rect x="1" y="3" width="6" height="6" rx="1" stroke="currentColor" strokeWidth="1.2" />
<rect x="9" y="7" width="6" height="6" rx="1" stroke="currentColor" strokeWidth="1.2" />
<path
d="M7 6L9 10"
stroke="currentColor"
strokeWidth="1.2"
strokeLinecap="round"
strokeDasharray="1.5 1.5"
/>
<circle cx="4" cy="6" r="1.5" stroke="currentColor" strokeWidth="1" />
</svg>
)
}

View File

@@ -0,0 +1,15 @@
export function IteratorIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M4 4h8M4 8h8M4 12h8" stroke="currentColor" strokeWidth="1.2" strokeLinecap="round" />
<path
d="M12 2l2 2-2 2M12 6l2 2-2 2M12 10l2 2-2 2"
stroke="currentColor"
strokeWidth="1"
strokeLinecap="round"
strokeLinejoin="round"
opacity="0.5"
/>
</svg>
)
}

View File

@@ -0,0 +1,15 @@
export function LoadImageIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<rect x="2" y="3" width="12" height="10" rx="1.5" stroke="currentColor" strokeWidth="1.33" />
<circle cx="5.5" cy="6.5" r="1.25" stroke="currentColor" strokeWidth="1.1" />
<path
d="M2 11L5.5 8L8 10L10.5 7.5L14 11"
stroke="currentColor"
strokeWidth="1.33"
strokeLinecap="round"
strokeLinejoin="round"
/>
</svg>
)
}

View File

@@ -0,0 +1,20 @@
export function ModelIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<path
d="M8 2L13 5V11L8 14L3 11V5L8 2Z"
stroke="currentColor"
strokeWidth="1.33"
strokeLinecap="round"
strokeLinejoin="round"
/>
<path
d="M8 8L13 5M8 8L3 5M8 8V14"
stroke="currentColor"
strokeWidth="1.33"
strokeLinecap="round"
strokeLinejoin="round"
/>
</svg>
)
}

View File

@@ -0,0 +1,13 @@
export function NumberIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<path
d="M5 2L4 14M12 2L11 14M2 5.5H14M2 10.5H14"
stroke="currentColor"
strokeWidth="1.33"
strokeLinecap="round"
strokeLinejoin="round"
/>
</svg>
)
}

View File

@@ -0,0 +1,7 @@
export function PlayIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M5 3L13 8L5 13V3Z" fill="currentColor" />
</svg>
)
}

View File

@@ -0,0 +1,14 @@
export function PreviewIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<path
d="M2 8C2 8 4.5 3 8 3C11.5 3 14 8 14 8C14 8 11.5 13 8 13C4.5 13 2 8 2 8Z"
stroke="currentColor"
strokeWidth="1.33"
strokeLinecap="round"
strokeLinejoin="round"
/>
<circle cx="8" cy="8" r="2" stroke="currentColor" strokeWidth="1.33" />
</svg>
)
}

View File

@@ -0,0 +1,11 @@
export function PromptConcatIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<rect x="1" y="2" width="5" height="3" rx="0.5" fill="currentColor" opacity="0.5" />
<rect x="1" y="6.5" width="5" height="3" rx="0.5" fill="currentColor" opacity="0.5" />
<rect x="1" y="11" width="5" height="3" rx="0.5" fill="currentColor" opacity="0.5" />
<path d="M7.5 3.5L9 8L7.5 12.5" stroke="currentColor" strokeWidth="1.2" />
<rect x="10" y="5.5" width="5" height="5" rx="0.5" fill="currentColor" />
</svg>
)
}

View File

@@ -0,0 +1,13 @@
export function PromptIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<path
d="M3 4H13M3 8H10M3 12H7"
stroke="currentColor"
strokeWidth="1.33"
strokeLinecap="round"
strokeLinejoin="round"
/>
</svg>
)
}

View File

@@ -0,0 +1,11 @@
export function RouterIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<circle cx="4" cy="8" r="2" fill="currentColor" />
<circle cx="12" cy="3" r="1.5" fill="currentColor" />
<circle cx="12" cy="8" r="1.5" fill="currentColor" />
<circle cx="12" cy="13" r="1.5" fill="currentColor" />
<path d="M6 8L10.5 3M6 8L10.5 8M6 8L10.5 13" stroke="currentColor" strokeWidth="1.2" />
</svg>
)
}

View File

@@ -0,0 +1,7 @@
export function StopIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<rect x="4" y="4" width="8" height="8" rx="1" fill="currentColor" />
</svg>
)
}

View File

@@ -0,0 +1,21 @@
export function StyleTransferIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<rect x="1" y="2" width="6" height="6" rx="1" stroke="currentColor" strokeWidth="1.2" />
<rect x="9" y="8" width="6" height="6" rx="1" stroke="currentColor" strokeWidth="1.2" />
<path
d="M7 5H9.5C10.3 5 11 5.7 11 6.5V8"
stroke="currentColor"
strokeWidth="1.2"
strokeLinecap="round"
/>
<path
d="M9.5 6.5L11 8L12.5 6.5"
stroke="currentColor"
strokeWidth="1.2"
strokeLinecap="round"
strokeLinejoin="round"
/>
</svg>
)
}

View File

@@ -0,0 +1,7 @@
export function SubtractIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<path d="M3 8H13" stroke="currentColor" strokeWidth="1.5" strokeLinecap="round" />
</svg>
)
}

View File

@@ -0,0 +1,19 @@
export function TemplateIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<rect
x="2"
y="2"
width="12"
height="12"
rx="2"
stroke="currentColor"
strokeWidth="1.2"
strokeDasharray="2 2"
/>
<rect x="4.5" y="4.5" width="3" height="3" rx="0.5" fill="currentColor" opacity="0.6" />
<rect x="8.5" y="4.5" width="3" height="3" rx="0.5" fill="currentColor" opacity="0.6" />
<rect x="6.5" y="8.5" width="3" height="3" rx="0.5" fill="currentColor" opacity="0.6" />
</svg>
)
}

View File

@@ -0,0 +1,15 @@
export function UpscaleIcon() {
return (
<svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
<rect x="1" y="5" width="6" height="6" rx="1" stroke="currentColor" strokeWidth="1.2" />
<rect x="6" y="1" width="9" height="9" rx="1" stroke="currentColor" strokeWidth="1.2" />
<path
d="M9 7L11.5 4.5M11.5 4.5H9M11.5 4.5V7"
stroke="currentColor"
strokeWidth="1.2"
strokeLinecap="round"
strokeLinejoin="round"
/>
</svg>
)
}

View File

@@ -0,0 +1,186 @@
import {
BindingOnChangeOptions,
BindingOnCreateOptions,
BindingOnDeleteOptions,
BindingOnShapeDeleteOptions,
BindingOnShapeIsolateOptions,
BindingUtil,
createComputedCache,
Editor,
T,
TLBinding,
TLShapeId,
} from 'tldraw'
import { getNodePorts } from '../nodes/nodePorts'
import { NodeShape } from '../nodes/NodeShapeUtil'
import { onNodePortConnect, onNodePortDisconnect } from '../nodes/nodeTypes'
import { PortId } from '../ports/Port'
import { ConnectionShape } from './ConnectionShapeUtil'
const CONNECTION_TYPE = 'connection'
declare module 'tldraw' {
export interface TLGlobalBindingPropsMap {
[CONNECTION_TYPE]: {
portId: PortId
terminal: 'start' | 'end'
order?: number
}
}
}
export type ConnectionBinding = TLBinding<typeof CONNECTION_TYPE>
export class ConnectionBindingUtil extends BindingUtil<ConnectionBinding> {
static override type = CONNECTION_TYPE
static override props = {
portId: T.string,
terminal: T.literalEnum('start', 'end'),
order: T.number.optional(),
}
override getDefaultProps() {
return {}
}
onBeforeIsolateToShape({ binding }: BindingOnShapeIsolateOptions<ConnectionBinding>): void {
this.editor.deleteShapes([binding.fromId])
}
onBeforeDeleteToShape({ binding }: BindingOnShapeDeleteOptions<ConnectionBinding>): void {
this.editor.deleteShapes([binding.fromId])
}
onAfterCreate({ binding }: BindingOnCreateOptions<ConnectionBinding>): void {
const node = this.editor.getShape(binding.toId)
if (!node || !this.editor.isShapeOfType(node, 'node')) return
onNodePortConnect(this.editor, node, binding.props.portId)
}
onAfterChange({ bindingBefore, bindingAfter }: BindingOnChangeOptions<ConnectionBinding>): void {
if (
bindingBefore.props.portId !== bindingAfter.props.portId ||
bindingBefore.toId !== bindingAfter.toId
) {
const nodeBefore = this.editor.getShape(bindingBefore.toId)
const nodeAfter = this.editor.getShape(bindingAfter.toId)
if (
!nodeBefore ||
!nodeAfter ||
!this.editor.isShapeOfType(nodeBefore, 'node') ||
!this.editor.isShapeOfType(nodeAfter, 'node')
) {
return
}
onNodePortDisconnect(this.editor, nodeBefore, bindingBefore.props.portId)
onNodePortConnect(this.editor, nodeAfter, bindingAfter.props.portId)
}
}
onAfterDelete({ binding }: BindingOnDeleteOptions<ConnectionBinding>): void {
const node = this.editor.getShape(binding.toId)
if (!node || !this.editor.isShapeOfType(node, 'node')) return
onNodePortDisconnect(this.editor, node, binding.props.portId)
}
}
export interface ConnectionBindings {
start?: ConnectionBinding
end?: ConnectionBinding
}
export function getConnectionBindings(
editor: Editor,
shape: ConnectionShape | TLShapeId
): ConnectionBindings {
return connectionBindingsCache.get(editor, typeof shape === 'string' ? shape : shape.id) ?? {}
}
const connectionBindingsCache = createComputedCache(
'connection bindings',
(editor: Editor, connection: ConnectionShape) => {
const bindings = editor.getBindingsFromShape(connection.id, CONNECTION_TYPE)
let start, end
for (const binding of bindings) {
if (binding.props.terminal === 'start') {
start = binding
} else if (binding.props.terminal === 'end') {
end = binding
}
}
return { start, end }
},
{
areRecordsEqual: (a, b) => a.id === b.id,
areResultsEqual: (a, b) => a.start === b.start && a.end === b.end,
}
)
export function getConnectionBindingPositionInPageSpace(
editor: Editor,
binding: ConnectionBinding
) {
const targetShape = editor.getShape(binding.toId)
if (!targetShape || !editor.isShapeOfType(targetShape, 'node')) return null
const port = getNodePorts(editor, targetShape)?.[binding.props.portId]
if (!port) return null
return editor.getShapePageTransform(targetShape).applyToPoint(port)
}
export function createOrUpdateConnectionBinding(
editor: Editor,
connection: ConnectionShape | TLShapeId,
target: NodeShape | TLShapeId,
props: ConnectionBinding['props']
) {
const connectionId = typeof connection === 'string' ? connection : connection.id
const targetId = typeof target === 'string' ? target : target.id
const existingMany = editor
.getBindingsFromShape(connectionId, CONNECTION_TYPE)
.filter((b) => b.props.terminal === props.terminal)
if (existingMany.length > 1) {
editor.deleteBindings(existingMany.slice(1))
}
// Auto-assign order for end-terminal (input port) bindings when not explicitly provided
let order = props.order
if (props.terminal === 'end' && order == null) {
const existingBindingsToTarget = editor
.getBindingsToShape(targetId, CONNECTION_TYPE)
.filter((b) => b.props.terminal === 'end' && b.props.portId === props.portId)
order = existingBindingsToTarget.reduce((max, b) => Math.max(max, b.props.order ?? 0), 0) + 1
}
const propsWithOrder = { ...props, order: order ?? props.order ?? 0 }
const existing = existingMany[0]
if (existing) {
editor.updateBinding({
...existing,
toId: targetId,
props: { ...propsWithOrder, order: existing.props.order ?? propsWithOrder.order },
})
} else {
editor.createBinding({
type: CONNECTION_TYPE,
fromId: connectionId,
toId: targetId,
props: propsWithOrder,
})
}
}
export function removeConnectionBinding(
editor: Editor,
connection: ConnectionShape,
terminal: 'start' | 'end'
) {
const existing = editor
.getBindingsFromShape(connection, CONNECTION_TYPE)
.filter((b) => b.props.terminal === terminal)
editor.deleteBindings(existing)
}

View File

@@ -0,0 +1,124 @@
import {
Circle2d,
Geometry2d,
OverlayUtil,
TLCursorType,
TLOverlay,
TLPointerEventInfo,
TLShapeId,
} from 'tldraw'
import {
CONNECTION_CENTER_HANDLE_HOVER_SIZE_PX,
CONNECTION_CENTER_HANDLE_SIZE_PX,
} from '../constants'
import { ConnectionShape, getConnectionPageCenter } from './ConnectionShapeUtil'
import { insertNodeWithinConnection } from './insertNodeWithinConnection'
interface TLConnectionCenterHandleOverlay extends TLOverlay {
props: {
shapeId: TLShapeId
x: number
y: number
}
}
/**
* Renders a clickable "+" handle at the midpoint of each fully-bound connection
* so users can insert a new node into the middle of a connection. Replaces the
* old SVG indicator-based handle.
*/
export class ConnectionCenterHandleOverlayUtil extends OverlayUtil<TLConnectionCenterHandleOverlay> {
static override type = 'connection_center_handle'
override options = { zIndex: 150 }
override isActive(): boolean {
const editor = this.editor
if (editor.getInstanceState().isReadonly) return false
if (!editor.isIn('select.idle')) return false
return editor.getZoomLevel() > 0.5
}
override getOverlays(): TLConnectionCenterHandleOverlay[] {
const editor = this.editor
const overlays: TLConnectionCenterHandleOverlay[] = []
for (const renderingShape of editor.getRenderingShapes()) {
const shape = renderingShape.shape
if (!editor.isShapeOfType<ConnectionShape>(shape, 'connection')) continue
const center = getConnectionPageCenter(editor, shape)
if (!center) continue
overlays.push({
id: `connection_center_handle:${shape.id}`,
type: 'connection_center_handle',
props: { shapeId: shape.id, x: center.x, y: center.y },
})
}
return overlays
}
override getGeometry(overlay: TLConnectionCenterHandleOverlay): Geometry2d {
const radius = CONNECTION_CENTER_HANDLE_HOVER_SIZE_PX / 2 / this.editor.getZoomLevel()
return new Circle2d({
x: overlay.props.x - radius,
y: overlay.props.y - radius,
radius,
isFilled: true,
})
}
override getCursor(): TLCursorType {
return 'pointer' as TLCursorType
}
override onPointerDown(overlay: TLConnectionCenterHandleOverlay, _info: TLPointerEventInfo) {
const connection = this.editor.getShape<ConnectionShape>(overlay.props.shapeId)
if (!connection) return false
insertNodeWithinConnection(this.editor, connection)
}
override render(
ctx: CanvasRenderingContext2D,
overlays: TLConnectionCenterHandleOverlay[]
): void {
if (overlays.length === 0) return
const editor = this.editor
const zoom = editor.getZoomLevel()
const themeColors = editor.getCurrentTheme().colors[editor.getColorMode()]
const ringColor = themeColors.selectionStroke
const iconColor = themeColors.selectedContrast
const hoverFill = themeColors.selectionFill
const hoveredId = editor.overlays.getHoveredOverlayId()
const ringRadius = CONNECTION_CENTER_HANDLE_SIZE_PX / 2 / zoom
const hoverRadius = CONNECTION_CENTER_HANDLE_HOVER_SIZE_PX / 2 / zoom
const plusReachPage = (CONNECTION_CENTER_HANDLE_SIZE_PX / 3 - 1) / zoom
const iconLineWidth = 2 / zoom
for (const overlay of overlays) {
const { x, y } = overlay.props
const isHovered = overlay.id === hoveredId
if (isHovered) {
ctx.fillStyle = hoverFill
ctx.beginPath()
ctx.arc(x, y, hoverRadius, 0, Math.PI * 2)
ctx.fill()
}
ctx.fillStyle = ringColor
ctx.beginPath()
ctx.arc(x, y, ringRadius, 0, Math.PI * 2)
ctx.fill()
ctx.strokeStyle = iconColor
ctx.lineWidth = iconLineWidth
ctx.lineCap = 'round'
ctx.beginPath()
ctx.moveTo(x - plusReachPage, y)
ctx.lineTo(x + plusReachPage, y)
ctx.moveTo(x, y - plusReachPage)
ctx.lineTo(x, y + plusReachPage)
ctx.stroke()
}
}
}

View File

@@ -0,0 +1,404 @@
import classNames from 'classnames'
import {
CubicBezier2d,
Editor,
IndexKey,
Mat,
RecordProps,
SVGContainer,
ShapeUtil,
TLHandle,
TLHandleDragInfo,
TLShape,
TLShapeId,
Vec,
VecLike,
VecModel,
clamp,
createShapeId,
useEditor,
useValue,
vecModelValidator,
} from 'tldraw'
import { onCanvasNodePickerState } from '../components/OnCanvasNodePicker'
import { PORT_TYPE_COLORS, PortDataType } from '../constants'
import {
getAllConnectedNodes,
getNodeOutputPortInfo,
getNodePorts,
getPortDataType,
} from '../nodes/nodePorts'
import { STOP_EXECUTION } from '../nodes/types/shared'
import { getPortAtPoint } from '../ports/getPortAtPoint'
import { findFirstCompatiblePort } from '../ports/portCompatibility'
import { updatePortState } from '../ports/portState'
import {
createOrUpdateConnectionBinding,
getConnectionBindingPositionInPageSpace,
getConnectionBindings,
removeConnectionBinding,
} from './ConnectionBindingUtil'
const CONNECTION_TYPE = 'connection'
declare module 'tldraw' {
export interface TLGlobalShapePropsMap {
[CONNECTION_TYPE]: {
start: VecModel
end: VecModel
}
}
}
export type ConnectionShape = TLShape<typeof CONNECTION_TYPE>
export class ConnectionShapeUtil extends ShapeUtil<ConnectionShape> {
static override type = CONNECTION_TYPE
static override props: RecordProps<ConnectionShape> = {
start: vecModelValidator,
end: vecModelValidator,
}
/** Connection ID that will be replaced if the current drag completes on an occupied port. */
private pendingReplacementId: TLShapeId | null = null
getDefaultProps(): ConnectionShape['props'] {
return {
start: { x: 0, y: 0 },
end: { x: 100, y: 100 },
}
}
override canEdit(_shape: ConnectionShape) {
return false
}
override canResize(_shape: ConnectionShape) {
return false
}
override hideResizeHandles(_shape: ConnectionShape) {
return true
}
override hideRotateHandle(_shape: ConnectionShape) {
return true
}
override hideSelectionBoundsBg(_shape: ConnectionShape) {
return true
}
override hideSelectionBoundsFg(_shape: ConnectionShape) {
return true
}
override canSnap(_shape: ConnectionShape) {
return false
}
override getBoundsSnapGeometry(_shape: ConnectionShape) {
return {
points: [],
}
}
getGeometry(connection: ConnectionShape) {
const { start, end } = getConnectionTerminals(this.editor, connection)
const [cp1, cp2] = getConnectionControlPoints(start, end)
return new CubicBezier2d({
start: Vec.From(start),
cp1: Vec.From(cp1),
cp2: Vec.From(cp2),
end: Vec.From(end),
})
}
getHandles(connection: ConnectionShape): TLHandle[] {
const { start, end } = getConnectionTerminals(this.editor, connection)
return [
{
id: 'start',
type: 'vertex',
index: 'a0' as IndexKey,
x: start.x,
y: start.y,
},
{
id: 'end',
type: 'vertex',
index: 'a1' as IndexKey,
x: end.x,
y: end.y,
},
]
}
onHandleDrag(connection: ConnectionShape, { handle }: TLHandleDragInfo<ConnectionShape>) {
const existingBindings = getConnectionBindings(this.editor, connection)
const draggingTerminal = handle.id as 'start' | 'end'
const oppositeTerminal = draggingTerminal === 'start' ? 'end' : 'start'
const oppositeTerminalShapeId = existingBindings[oppositeTerminal]?.toId
const shapeTransform = this.editor.getShapePageTransform(connection)
const handlePagePosition = shapeTransform.applyToPoint(handle)
const target = getPortAtPoint(this.editor, handlePagePosition, {
margin: 8,
terminal: handle.id as 'start' | 'end',
})
const existingConnectionOnTarget =
target?.existingConnections.find((c) => c.connectionId !== connection.id) ?? null
const nodesWhichWouldCreateACycle = oppositeTerminalShapeId
? getAllConnectedNodes(this.editor, oppositeTerminalShapeId, draggingTerminal)
: null
// Determine the data type of the opposite end for type-checking
const oppositeBinding = existingBindings[oppositeTerminal]
let dragDataType: PortDataType | null = null
if (oppositeBinding) {
dragDataType = getPortDataType(
this.editor,
oppositeBinding.toId,
oppositeBinding.props.portId
)
}
updatePortState(this.editor, {
eligiblePorts: {
terminal: draggingTerminal,
excludeNodes: nodesWhichWouldCreateACycle,
dataType: dragDataType,
},
})
// Check type compatibility
const isTypeIncompatible =
target &&
dragDataType &&
dragDataType !== 'any' &&
target.port.dataType !== 'any' &&
target.port.dataType !== dragDataType
const wouldCreateACycle = (target && nodesWhichWouldCreateACycle?.has(target.shape.id)) ?? false
if (!target || wouldCreateACycle || isTypeIncompatible) {
this.pendingReplacementId = null
updatePortState(this.editor, { hintingPort: null })
removeConnectionBinding(this.editor, connection, draggingTerminal)
return {
...connection,
props: {
[handle.id]: { x: handle.x, y: handle.y },
},
}
}
// Track the connection that would be replaced, but don't delete it yet.
// Multi-ports accept multiple connections, so skip replacement for them.
this.pendingReplacementId =
existingConnectionOnTarget && draggingTerminal === 'end' && !target.port.multi
? existingConnectionOnTarget.connectionId
: null
updatePortState(this.editor, {
hintingPort: { portId: target.port.id, shapeId: target.shape.id },
})
createOrUpdateConnectionBinding(this.editor, connection, target.shape, {
portId: target.port.id,
terminal: draggingTerminal,
})
return connection
}
onHandleDragEnd(
connection: ConnectionShape,
{ handle, isCreatingShape }: TLHandleDragInfo<ConnectionShape>
) {
// Delete the connection being replaced now that the drag is committed.
if (this.pendingReplacementId) {
this.editor.deleteShapes([this.pendingReplacementId])
this.pendingReplacementId = null
}
updatePortState(this.editor, { hintingPort: null, eligiblePorts: null })
const draggingTerminal = handle.id as 'start' | 'end'
const bindings = getConnectionBindings(this.editor, connection)
if (bindings[draggingTerminal]) {
return
}
if (isCreatingShape && draggingTerminal === 'end') {
this.editor.selectNone()
onCanvasNodePickerState.set(this.editor, {
connectionShapeId: connection.id,
location: draggingTerminal,
onClose: () => {
const bindings = getConnectionBindings(this.editor, connection)
if (!bindings.start || !bindings.end) {
this.editor.deleteShapes([connection.id])
}
},
onPick: (nodeType, terminalInPageSpace) => {
const newNodeId = createShapeId()
this.editor.createShape({
type: 'node',
id: newNodeId,
x: terminalInPageSpace.x,
y: terminalInPageSpace.y,
props: {
node: nodeType,
},
})
this.editor.select(newNodeId)
const bindings = getConnectionBindings(this.editor, connection)
const sourceType = bindings.start
? (getPortDataType(this.editor, bindings.start.toId, bindings.start.props.portId) ??
'any')
: 'any'
const ports = getNodePorts(this.editor, newNodeId)
const firstCompatibleInputPort = findFirstCompatiblePort(
Object.values(ports),
'end',
sourceType
)
if (firstCompatibleInputPort) {
this.editor.updateShape({
id: newNodeId,
type: 'node',
x: terminalInPageSpace.x - firstCompatibleInputPort.x,
y: terminalInPageSpace.y - firstCompatibleInputPort.y,
})
createOrUpdateConnectionBinding(this.editor, connection, newNodeId, {
portId: firstCompatibleInputPort.id,
terminal: draggingTerminal,
})
}
},
})
} else {
if (!bindings.start || !bindings.end) {
this.editor.deleteShapes([connection.id])
}
}
}
onHandleDragCancel() {
this.pendingReplacementId = null
updatePortState(this.editor, { hintingPort: null, eligiblePorts: null })
}
component(connection: ConnectionShape) {
return <ConnectionShapeComponent connection={connection} />
}
getIndicatorPath(connection: ConnectionShape) {
const { start, end } = getConnectionTerminals(this.editor, connection)
return new Path2D(getConnectionPath(start, end))
}
}
function ConnectionShapeComponent({ connection }: { connection: ConnectionShape }) {
const editor = useEditor()
const { start, end } = useValue('terminals', () => getConnectionTerminals(editor, connection), [
editor,
connection,
])
// Get the data type color for this connection from its start binding
const connectionColor = useValue(
'connectionColor',
() => {
const bindings = getConnectionBindings(editor, connection.id)
if (!bindings.start) return null
const dataType = getPortDataType(editor, bindings.start.toId, bindings.start.props.portId)
return dataType ? PORT_TYPE_COLORS[dataType] : null
},
[connection.id, editor]
)
const isInactive = useValue(
'isInactive',
() => {
const bindings = getConnectionBindings(editor, connection.id)
if (!bindings.start) return false
const originShapeId = bindings.start?.toId
if (!originShapeId) return false
const outputs = getNodeOutputPortInfo(editor, originShapeId)
const output = outputs[bindings.start.props.portId]
return output?.value === STOP_EXECUTION
},
[connection.id, editor]
)
return (
<SVGContainer
className={classNames('ConnectionShape', isInactive && 'ConnectionShape_inactive')}
>
<path
d={getConnectionPath(start, end)}
style={connectionColor ? { stroke: connectionColor } : undefined}
/>
</SVGContainer>
)
}
export function getConnectionControlPoints(start: VecLike, end: VecLike): [Vec, Vec] {
const distance = end.x - start.x
const adjustedDistance = Math.max(
30,
distance > 0 ? distance / 3 : clamp(Math.abs(distance) + 30, 0, 100)
)
return [new Vec(start.x + adjustedDistance, start.y), new Vec(end.x - adjustedDistance, end.y)]
}
/**
* Page-space midpoint of a connection's bezier curve, for positioning the
* center insert handle. Returns null when the connection isn't fully bound.
*/
export function getConnectionPageCenter(editor: Editor, connection: ConnectionShape): Vec | null {
const bindings = getConnectionBindings(editor, connection)
if (!bindings.start || !bindings.end) return null
const startPage = getConnectionBindingPositionInPageSpace(editor, bindings.start)
const endPage = getConnectionBindingPositionInPageSpace(editor, bindings.end)
if (!startPage || !endPage) return null
const [cp1, cp2] = getConnectionControlPoints(startPage, endPage)
// Cubic bezier midpoint at t=0.5: (P0 + 3·P1 + 3·P2 + P3) / 8
return new Vec(
(startPage.x + 3 * cp1.x + 3 * cp2.x + endPage.x) / 8,
(startPage.y + 3 * cp1.y + 3 * cp2.y + endPage.y) / 8
)
}
function getConnectionPath(start: VecLike, end: VecLike) {
const [cp1, cp2] = getConnectionControlPoints(start, end)
return `M ${start.x} ${start.y} C ${cp1.x} ${cp1.y} ${cp2.x} ${cp2.y} ${end.x} ${end.y}`
}
export function getConnectionTerminals(editor: Editor, connection: ConnectionShape) {
let start, end
const bindings = getConnectionBindings(editor, connection)
const shapeTransform = Mat.Inverse(editor.getShapePageTransform(connection))
if (bindings.start) {
const inPageSpace = getConnectionBindingPositionInPageSpace(editor, bindings.start)
if (inPageSpace) {
start = Mat.applyToPoint(shapeTransform, inPageSpace)
}
}
if (bindings.end) {
const inPageSpace = getConnectionBindingPositionInPageSpace(editor, bindings.end)
if (inPageSpace) {
end = Mat.applyToPoint(shapeTransform, inPageSpace)
}
}
if (!start) start = connection.props.start
if (!end) end = connection.props.end
return { start, end }
}

View File

@@ -0,0 +1,87 @@
import { createShapeId, Editor } from 'tldraw'
import { onCanvasNodePickerState } from '../components/OnCanvasNodePicker'
import { DEFAULT_NODE_SPACING_PX, NODE_WIDTH_PX } from '../constants'
import { getNodePorts, getPortDataType } from '../nodes/nodePorts'
import { findFirstCompatiblePort } from '../ports/portCompatibility'
import { createOrUpdateConnectionBinding, getConnectionBindings } from './ConnectionBindingUtil'
import { ConnectionShape } from './ConnectionShapeUtil'
/**
* Insert a node in the middle of a connection.
*/
export function insertNodeWithinConnection(editor: Editor, connection: ConnectionShape) {
onCanvasNodePickerState.set(editor, {
connectionShapeId: connection.id,
location: 'middle',
onPick: (nodeType) => {
const mark = editor.markHistoryStoppingPoint()
const originalBindings = getConnectionBindings(editor, connection)
if (!originalBindings.start || !originalBindings.end) return
const startBounds = editor.getShapePageBounds(originalBindings.start.toId)!
const endBounds = editor.getShapePageBounds(originalBindings.end.toId)!
const newNodeY = (startBounds.top + endBounds.top) / 2
const newNodeIdealX = (startBounds.right + endBounds.left - NODE_WIDTH_PX) / 2
const newNodeMin = startBounds.right + DEFAULT_NODE_SPACING_PX
const newNodeX = Math.max(newNodeIdealX, newNodeMin)
const newNodeId = createShapeId()
editor.createShape({
type: 'node',
id: newNodeId,
x: newNodeX,
y: newNodeY,
props: { node: nodeType },
})
const sourceType = getPortDataType(
editor,
originalBindings.start.toId,
originalBindings.start.props.portId
)
const targetType = getPortDataType(
editor,
originalBindings.end.toId,
originalBindings.end.props.portId
)
const ports = getNodePorts(editor, newNodeId)
const firstCompatibleInputPort = sourceType
? findFirstCompatiblePort(Object.values(ports), 'end', sourceType)
: Object.values(ports).find((p) => p.terminal === 'end')
const firstCompatibleOutputPort = targetType
? findFirstCompatiblePort(Object.values(ports), 'start', targetType)
: Object.values(ports).find((p) => p.terminal === 'start')
if (!firstCompatibleInputPort || !firstCompatibleOutputPort) {
editor.bailToMark(mark)
return
}
createOrUpdateConnectionBinding(editor, connection, newNodeId, {
portId: firstCompatibleInputPort.id,
terminal: 'end',
})
const newConnectionId = createShapeId()
editor.createShape({
type: 'connection',
id: newConnectionId,
})
createOrUpdateConnectionBinding(editor, newConnectionId, newNodeId, {
portId: firstCompatibleOutputPort.id,
terminal: 'start',
})
createOrUpdateConnectionBinding(editor, newConnectionId, originalBindings.end.toId, {
portId: originalBindings.end.props.portId,
terminal: 'end',
})
editor.select(newNodeId)
editor.updatePointer()
},
onClose: () => {},
})
}

View File

@@ -0,0 +1,95 @@
import { Editor, getIndexBetween, getIndicesBetween, TLParentId } from 'tldraw'
export function keepConnectionsAtBottom(editor: Editor) {
let pendingChangedParentIds = new Set<TLParentId>()
editor.sideEffects.registerAfterCreateHandler('shape', (shape, source) => {
if (source === 'remote') return
pendingChangedParentIds.add(shape.parentId)
})
editor.sideEffects.registerAfterChangeHandler('shape', (oldShape, newShape, source) => {
if (source === 'remote') return
if (oldShape.parentId === newShape.parentId && oldShape.index === newShape.index) return
pendingChangedParentIds.add(newShape.parentId)
})
editor.sideEffects.registerOperationCompleteHandler(() => {
if (pendingChangedParentIds.size === 0) return
const changedParentIds = pendingChangedParentIds
pendingChangedParentIds = new Set()
const updates = []
for (const parentId of changedParentIds) {
const childIds = editor.getSortedChildIdsForParent(parentId)
let i = childIds.length - 1
let highestConnectionIndex = null
let nextIndexAboveHighestConnectionIndex = null
for (; i >= 0; i--) {
const child = editor.getShape(childIds[i])
if (!child) continue
if (child.type === 'connection') {
highestConnectionIndex = child.index
break
} else {
nextIndexAboveHighestConnectionIndex = child.index
}
}
const shapesToMove = []
for (; i >= 0; i--) {
const child = editor.getShape(childIds[i])
if (!child) continue
if (child.type !== 'connection') {
shapesToMove.push(child)
}
}
shapesToMove.reverse()
const newIndexes = getIndicesBetween(
highestConnectionIndex,
nextIndexAboveHighestConnectionIndex,
shapesToMove.length
)
for (let i = 0; i < shapesToMove.length; i++) {
const shape = shapesToMove[i]
const newIndex = newIndexes[i]
updates.push({
id: shape.id,
type: shape.type,
index: newIndex,
} as const)
}
}
editor.updateShapes(updates)
})
}
export function getNextConnectionIndex(
editor: Editor,
parentId: TLParentId = editor.getCurrentPageId()
) {
const childIds = editor.getSortedChildIdsForParent(parentId)
let prevIndex = null
let highestConnectionIndex = null
for (let i = childIds.length - 1; i >= 0; i--) {
const child = editor.getShape(childIds[i])
if (!child) continue
if (child.type === 'connection') {
highestConnectionIndex = child.index
break
}
prevIndex = child.index
}
return getIndexBetween(highestConnectionIndex, prevIndex)
}

View File

@@ -0,0 +1,30 @@
export const CONNECTION_CENTER_HANDLE_SIZE_PX = 16
export const CONNECTION_CENTER_HANDLE_HOVER_SIZE_PX = 24
export const NODE_WIDTH_PX = 260
export const NODE_HEADER_HEIGHT_PX = 40
export const NODE_ROW_HEADER_GAP_PX = 8
export const NODE_ROW_BOTTOM_PADDING_PX = 8
export const NODE_FOOTER_HEIGHT_PX = 40
export const NODE_ROW_HEIGHT_PX = 44
export const NODE_IMAGE_PREVIEW_HEIGHT_PX = 160 + 8
export const PORT_RADIUS_PX = 6
export const DEFAULT_NODE_SPACING_PX = 60
/**
* Port data types define the kind of data that flows between nodes. Each type
* has a CSS color used to tint ports and connections so users can see at a
* glance which outputs are compatible with which inputs.
*/
export type PortDataType = 'image' | 'text' | 'model' | 'number' | 'latent' | 'any'
export const PORT_TYPE_COLORS: Record<PortDataType, string> = {
image: '#c060e0',
text: '#4caf50',
model: '#2196f3',
number: '#9e9e9e',
latent: '#ff9800',
any: '#c08520',
}

View File

@@ -0,0 +1,17 @@
import { Editor } from 'tldraw'
export function disableTransparency(editor: Editor, shapeTypes: string[]) {
const shapeTypesSet = new Set(shapeTypes)
editor.sideEffects.registerBeforeCreateHandler('shape', (shape) => {
if (!shapeTypesSet.has(shape.type)) return shape
if (shape.opacity === 1) return shape
return { ...shape, opacity: 1 }
})
editor.sideEffects.registerBeforeChangeHandler('shape', (_shapeOld, shapeNew) => {
if (!shapeTypesSet.has(shapeNew.type)) return shapeNew
if (shapeNew.opacity === 1) return shapeNew
return { ...shapeNew, opacity: 1 }
})
}

View File

@@ -0,0 +1,166 @@
import { AtomMap, Editor, TLShapeId } from 'tldraw'
import {
getNodeOutputPortInfo,
getNodePortConnections,
getNodePorts,
NodePortConnection,
} from '../nodes/nodePorts'
import { NodeShape } from '../nodes/NodeShapeUtil'
import { executeNode } from '../nodes/nodeTypes'
import { ExecutionResult, PipelineValue, STOP_EXECUTION } from '../nodes/types/shared'
interface PendingExecutionGraphNode {
readonly state: 'waiting' | 'executing'
readonly shape: NodeShape
readonly connections: NodePortConnection[]
}
interface ExecutedExecutionGraphNode {
readonly state: 'executed'
readonly shape: NodeShape
readonly connections: NodePortConnection[]
readonly outputs: ExecutionResult
}
type ExecutionGraphNode = PendingExecutionGraphNode | ExecutedExecutionGraphNode
export class ExecutionGraph {
private readonly nodesById = new AtomMap<TLShapeId, ExecutionGraphNode>('node by id')
constructor(
private readonly editor: Editor,
private readonly startingNodeIds: Set<TLShapeId>
) {
const toVisit = Array.from(startingNodeIds)
while (toVisit.length > 0) {
const nodeId = toVisit.pop()!
if (this.nodesById.has(nodeId)) continue
const node = this.editor.getShape(nodeId)
if (!node || !this.editor.isShapeOfType(node, 'node')) continue
const connections = getNodePortConnections(this.editor, node)
this.nodesById.set(nodeId, {
state: 'waiting',
shape: node,
connections,
})
for (const connection of Object.values(connections)) {
if (!connection || connection.terminal !== 'start') continue
toVisit.push(connection.connectedShapeId)
}
}
}
private state: 'waiting' | 'executing' | 'stopped' = 'waiting'
async execute() {
if (this.state !== 'waiting') {
throw new Error('ExecutionGraph can only be executed once')
}
this.state = 'executing'
try {
const promises = []
for (const nodeId of this.startingNodeIds) {
promises.push(this.executeNodeIfReady(nodeId))
}
await Promise.all(promises)
} finally {
this.state = 'stopped'
}
}
stop() {
this.state = 'stopped'
}
private async executeNodeIfReady(nodeId: TLShapeId) {
if (this.state !== 'executing') return
const node = this.nodesById.get(nodeId)
if (!node || node.state !== 'waiting') return
const inputs: Record<string, PipelineValue | PipelineValue[]> = {}
const ports = getNodePorts(this.editor, nodeId)
const sortedConnections = [...node.connections].sort((a, b) => a.order - b.order)
for (const connection of sortedConnections) {
if (!connection || connection.terminal !== 'end') continue
const dependency = this.nodesById.get(connection.connectedShapeId)
let value: PipelineValue | STOP_EXECUTION
if (dependency) {
if (dependency.state !== 'executed') {
return
}
const output = dependency.outputs[connection.connectedPortId]
if (output === STOP_EXECUTION) {
return
}
value = output as PipelineValue
} else {
const outputs = getNodeOutputPortInfo(this.editor, connection.connectedShapeId)
const output = outputs[connection.connectedPortId]
if (!output || output.value === STOP_EXECUTION) {
return
}
value = output.value as PipelineValue
}
const port = ports[connection.ownPortId]
if (port?.multi) {
const existing = inputs[connection.ownPortId]
if (Array.isArray(existing)) {
existing.push(value)
} else {
inputs[connection.ownPortId] = [value]
}
} else {
inputs[connection.ownPortId] = value
}
}
this.nodesById.set(nodeId, {
...node,
state: 'executing',
})
this.editor.updateShape({
id: nodeId,
type: node.shape.type,
props: { isOutOfDate: true },
})
const outputs = await executeNode(this.editor, node.shape, inputs)
this.editor.updateShape({
id: nodeId,
type: node.shape.type,
props: { isOutOfDate: false },
})
this.nodesById.set(nodeId, {
...node,
state: 'executed',
outputs,
})
const executingDependentPromises = []
for (const connection of Object.values(node.connections)) {
if (!connection || connection.terminal !== 'start') continue
executingDependentPromises.push(this.executeNodeIfReady(connection.connectedShapeId))
}
await Promise.all(executingDependentPromises)
}
getNodeStatus(nodeId: TLShapeId) {
return this.nodesById.get(nodeId)?.state
}
}

View File

@@ -0,0 +1,38 @@
import { Editor, TLShapeId } from 'tldraw'
import { EditorAtom } from '../utils'
import { ExecutionGraph } from './ExecutionGraph'
export interface ExecutionState {
runningGraph: ExecutionGraph | null
}
export const executionState = new EditorAtom<ExecutionState>('execution state', () => ({
runningGraph: null,
}))
export async function startExecution(editor: Editor, startingNodeIds: Set<TLShapeId>) {
const graph = new ExecutionGraph(editor, startingNodeIds)
executionState.update(editor, (state) => {
state.runningGraph?.stop()
return {
...state,
runningGraph: graph,
}
})
try {
await graph.execute()
} finally {
executionState.update(editor, (state) => {
if (state.runningGraph !== graph) return state
return { ...state, runningGraph: null }
})
}
}
export function stopExecution(editor: Editor) {
executionState.update(editor, (state) => {
if (!state.runningGraph) return state
state.runningGraph.stop()
return { ...state, runningGraph: null }
})
}

View File

@@ -0,0 +1,833 @@
@import url('https://fonts.googleapis.com/css2?family=Inter:wght@400;500;700&display=swap');
@import url('tldraw/tldraw.css');
body {
font-family: 'Inter', sans-serif;
overscroll-behavior: none;
}
.tl-container {
--node-shadows: 0 2px 6px rgba(0, 0, 0, 0.08), 0 1px 2px rgba(0, 0, 0, 0.04);
--node-shadow-border:
0 0 0 1px rgba(0, 0, 0, 0.06), 0 2px 6px rgba(0, 0, 0, 0.08), 0 1px 2px rgba(0, 0, 0, 0.04);
--node-shadow-highlight:
0 0 0 calc(1.2px / var(--tl-zoom)) var(--tl-color-selected), 0 2px 6px rgba(0, 0, 0, 0.08),
0 1px 2px rgba(0, 0, 0, 0.04);
/* Port type colors as CSS custom properties */
--port-color-image: #c060e0;
--port-color-text: #4caf50;
--port-color-model: #2196f3;
--port-color-number: #9e9e9e;
--port-color-latent: #ff9800;
--port-color-any: #b0b0b0;
}
.tl-container .tl-theme__light {
--tl-color-handle-fill: hsla(210, 100%, 56%, 0.24);
}
.tl-container .tl-theme__dark {
--tl-color-handle-fill: hsla(209, 100%, 57%, 0.2);
}
/* ---- Node shape ---- */
.NodeShape {
border-radius: var(--tl-radius-3);
background: var(--tl-color-panel);
box-shadow: var(--node-shadow-border);
width: 100%;
height: fit-content;
display: flex;
flex-direction: column;
.NodeShape-heading {
padding: 0 var(--tl-space-4);
margin-bottom: var(--tl-space-3);
display: flex;
align-items: center;
justify-content: center;
gap: var(--tl-space-3);
height: 40px;
border-bottom: 1px solid var(--tl-color-divider);
}
.NodeShape-icon {
flex: 0 0 auto;
display: flex;
align-items: center;
color: var(--tl-color-text-3);
}
.NodeShape-label {
flex: 1 1 auto;
font-weight: 500;
font-size: 13px;
}
.NodeShape-output {
font-variant-numeric: tabular-nums;
font-size: 11px;
color: var(--tl-color-text-3);
max-width: 80px;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
/* Enable pointer events on the entire node body so all interactive
elements work. The heading stays inert (drag handle for tldraw). */
[data-state='select.idle'] & > :not(.NodeShape-heading) {
pointer-events: all;
}
.NodeShape-footer {
display: flex;
align-items: center;
gap: var(--tl-space-2);
height: 40px;
padding: 0 var(--tl-space-1) 0 var(--tl-space-4);
border-top: 1px solid var(--tl-color-divider);
margin-top: var(--tl-space-3);
}
.NodeShape-footer-action {
display: flex;
align-items: center;
gap: var(--tl-space-2);
border: none;
background: none;
padding: 2px 6px;
margin: 0 -6px;
border-radius: var(--tl-radius-2);
font: inherit;
font-size: 11px;
color: var(--tl-color-text-3);
cursor: pointer;
svg {
width: 12px;
height: 12px;
}
&:hover {
background: var(--tl-color-muted-0);
color: var(--tl-color-text-1);
}
&.NodeShape-footer-action_executing {
color: var(--tl-color-selected);
}
}
.NodeFooterMenu {
margin-left: auto;
}
.NodeFooterMenu-trigger {
width: 32px;
min-width: 32px;
height: 32px;
flex-shrink: 0;
color: var(--tl-color-text-3);
}
&.NodeShape_executing {
box-shadow:
0 0 0 calc(2px / var(--tl-zoom)) var(--tl-color-selected),
0 2px 6px rgba(0, 0, 0, 0.08),
0 1px 2px rgba(0, 0, 0, 0.04);
}
}
/* ---- Capture node ---- */
.NodeShape_capture {
background: transparent;
box-shadow: none;
border: 2px dashed rgba(128, 128, 128, 0.4);
}
.NodeShape_capture .NodeShape-heading {
background: rgba(255, 255, 255, 0.85);
border-radius: var(--tl-radius-3) var(--tl-radius-3) 0 0;
}
.NodeShape_capture .NodeShape-footer {
background: rgba(255, 255, 255, 0.85);
border-radius: 0 0 var(--tl-radius-3) var(--tl-radius-3);
}
/* ---- Node rows ---- */
.NodeRow {
display: flex;
flex-direction: row;
align-items: center;
justify-content: flex-start;
height: 44px;
gap: var(--tl-space-3);
padding: var(--tl-space-2) var(--tl-space-4);
position: relative;
input[type='text'],
select {
align-self: stretch;
flex: 1 1 auto;
border: 1px solid var(--tl-color-hint);
padding: 0 var(--tl-space-4);
border-radius: var(--tl-radius-3);
background: none;
font: inherit;
font-size: 12px;
outline: none;
&:focus {
border-color: var(--tl-color-selected);
}
}
input[type='range'] {
flex: 1 1 auto;
accent-color: var(--tl-color-selected);
}
select {
-webkit-appearance: none;
-moz-appearance: none;
appearance: none;
background-image: url("data:image/svg+xml;utf8,<svg width='30' height='30' viewBox='0 0 30 30' fill='none' xmlns='http://www.w3.org/2000/svg'><path fill-rule='evenodd' clip-rule='evenodd' d='M6.27047 12.3161C6.6482 11.9131 7.28103 11.8927 7.68394 12.2705L15 19.1293L22.3161 12.2705C22.719 11.8927 23.3518 11.9131 23.7295 12.3161C24.1073 12.719 24.0869 13.3518 23.6839 13.7295L15.6839 21.2295C15.2993 21.5902 14.7007 21.5902 14.3161 21.2295L6.31606 13.7295C5.91315 13.3518 5.89274 12.719 6.27047 12.3161Z' fill='black'/></svg>");
background-repeat: no-repeat;
background-position: right var(--tl-space-4) center;
background-size: 16px;
}
textarea {
flex: 1 1 auto;
border: 1px solid var(--tl-color-hint);
padding: var(--tl-space-3) var(--tl-space-4);
border-radius: var(--tl-radius-3);
background: none;
font: inherit;
font-size: 12px;
outline: none;
resize: none;
&:focus {
border-color: var(--tl-color-selected);
}
}
}
.NodeInputRow {
.NodeInputRow-label {
font-size: 11px;
color: var(--tl-color-text-3);
flex: 0 0 auto;
min-width: 48px;
}
.NodeInputRow-buttons {
display: none;
position: absolute;
right: 19px;
button {
margin-right: -8px;
min-width: 38px;
height: 38px;
}
}
&:has(input[type='text']:not(:disabled)) {
&:hover,
&:has(input:focus) {
.NodeInputRow-buttons {
display: flex;
}
}
}
}
.NodeRow-connected-value {
font-size: 11px;
color: var(--tl-color-text-3);
flex: 1 1 auto;
text-align: right;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
.NodeRow-disconnected {
font-size: 11px;
color: var(--tl-color-text-3);
opacity: 0.5;
flex: 1 1 auto;
text-align: right;
font-style: italic;
}
.NodeRow-value {
font-size: 11px;
color: var(--tl-color-text-3);
font-variant-numeric: tabular-nums;
flex: 0 0 auto;
min-width: 32px;
text-align: right;
}
.NodePortLabel {
font-size: 11px;
font-weight: 500;
flex: 0 0 auto;
}
.NodeValue_placeholder {
display: inline-block;
width: 18px;
height: 10px;
border-radius: 10px;
background: hsl(0, 0%, 0%, 10%);
}
/* ---- Image previews ---- */
.NodeImagePreview {
margin: var(--tl-space-2) var(--tl-space-4);
height: 160px;
border-radius: var(--tl-radius-3);
overflow: hidden;
background-color: hsl(0 0% 96%);
background-image:
linear-gradient(45deg, hsl(0 0% 92%) 25%, transparent 25%, transparent 75%, hsl(0 0% 92%) 75%),
linear-gradient(45deg, hsl(0 0% 92%) 25%, transparent 25%, transparent 75%, hsl(0 0% 92%) 75%);
background-size: 12px 12px;
background-position:
0 0,
6px 6px;
border: 1px solid var(--tl-color-divider);
img {
width: 100%;
height: 100%;
object-fit: contain;
}
}
.NodeImagePreview_dragover {
outline: 2px solid var(--tl-color-selected);
outline-offset: -2px;
background: var(--tl-color-selected-0);
}
.NodeImagePreview_loading {
position: relative;
&::after {
content: '';
position: absolute;
inset: 0;
background: linear-gradient(
90deg,
transparent 0%,
rgba(255, 255, 255, 0.3) 50%,
transparent 100%
);
animation: shimmer 1.5s ease-in-out infinite;
}
}
@keyframes shimmer {
0% {
transform: translateX(-100%);
}
100% {
transform: translateX(100%);
}
}
/* ---- Load image node ---- */
.LoadImageNode-browse {
flex: 1 1 auto;
border: 1px solid var(--tl-color-hint);
border-radius: var(--tl-radius-3);
padding: var(--tl-space-2) var(--tl-space-3);
font: inherit;
font-size: 11px;
background: var(--tl-color-panel);
cursor: pointer;
&:hover {
background: var(--tl-color-muted-0);
}
}
.LoadImageNode-clear {
flex: 0 0 auto;
border: 1px solid var(--tl-color-hint);
border-radius: var(--tl-radius-3);
padding: var(--tl-space-2) var(--tl-space-3);
font: inherit;
font-size: 14px;
line-height: 1;
background: var(--tl-color-panel);
cursor: pointer;
color: var(--tl-color-text-3);
&:hover {
background: var(--tl-color-muted-0);
color: var(--tl-color-text-1);
}
}
.NodeImagePreview-empty {
width: 100%;
height: 100%;
display: flex;
align-items: center;
justify-content: center;
span {
font-size: 11px;
color: var(--tl-color-text-3);
opacity: 0.5;
}
}
/* ---- Prompt node ---- */
.PromptNode-row {
height: 88px;
align-items: stretch;
}
.PromptNode-textarea {
width: 100%;
height: 100%;
}
/* ---- Generate text node ---- */
.GenerateTextNode-result {
margin: var(--tl-space-2) var(--tl-space-4);
height: 88px;
border-radius: var(--tl-radius-3);
overflow: hidden;
background: var(--tl-color-muted-0);
border: 1px solid var(--tl-color-divider);
}
.GenerateTextNode-result_loading {
position: relative;
&::after {
content: '';
position: absolute;
inset: 0;
background: linear-gradient(
90deg,
transparent 0%,
rgba(255, 255, 255, 0.3) 50%,
transparent 100%
);
animation: shimmer 1.5s ease-in-out infinite;
}
}
.GenerateTextNode-result-text {
width: 100%;
height: 100%;
padding: var(--tl-space-3) var(--tl-space-4);
font-size: 11px;
line-height: 1.4;
color: var(--tl-color-text-1);
overflow-y: scroll;
white-space: pre-wrap;
word-break: break-word;
}
.GenerateTextNode-result-empty {
width: 100%;
height: 100%;
display: flex;
align-items: center;
justify-content: center;
span {
font-size: 11px;
color: var(--tl-color-text-3);
opacity: 0.5;
}
}
/* ---- Number node ---- */
.NumberNode {
.tlui-slider__container {
padding: 0;
&:has(.tlui-slider__thumb:focus-visible) {
outline: none;
}
}
}
/* ---- Ports ---- */
.Port {
position: absolute;
width: 12px;
height: 12px;
border-radius: 50%;
box-sizing: border-box;
z-index: 2;
flex: 0 0 auto;
transition: transform 0.05s ease-in-out;
/* Use port color from inline style or fall back to gray */
background: var(--port-color, var(--tl-color-panel));
box-shadow:
0 0 0 2px var(--tl-color-panel),
0 0 0 3px var(--port-color, rgba(0, 0, 0, 0.15));
[data-state='select.idle'] & {
pointer-events: all;
}
&.Port_hinting {
transform: scale(1.3);
box-shadow:
0 0 0 2px var(--tl-color-panel),
0 0 0 3px var(--port-color, var(--tl-color-selected)),
0 0 8px var(--port-color, var(--tl-color-selected));
}
&.Port_end {
left: -6px;
}
&.Port_start {
right: -6px;
}
&.Port_eligible {
box-shadow:
0 0 0 2px var(--tl-color-panel),
0 0 0 3px var(--port-color, rgba(0, 0, 0, 0.15)),
0 0 4px var(--port-color, var(--tl-color-selected));
}
}
/* ---- Connections ---- */
.ConnectionShape {
stroke-width: 2px;
stroke: var(--tl-color-text-3);
fill: none;
&.ConnectionShape_inactive {
opacity: 0.3;
stroke-dasharray: 4 4;
}
}
.ConnectionShapeIndicator {
stroke-width: 2.1px;
stroke-linecap: round;
}
/* ---- On-canvas node picker ---- */
.OnCanvasNodePicker {
position: absolute;
top: 0;
left: 0;
z-index: var(--tl-layer-overlays);
--top: 66px;
transform-origin: 0 0;
.OnCanvasNodePicker-content {
position: absolute;
margin-top: calc(-1 * var(--top));
width: 170px;
border-radius: var(--tl-radius-3);
background: var(--tl-color-panel);
box-shadow: var(--node-shadow-highlight);
}
&.OnCanvasNodePicker_middle .OnCanvasNodePicker-content {
margin-left: -85px;
}
}
/* ---- Pipeline regions ---- */
.tl-canvas__in-front {
z-index: var(--tl-layer-above);
}
.PipelineRegion {
position: absolute;
color: hsl(260, 40%, 70%);
border: 1px dashed currentColor;
border-radius: var(--tl-radius-3);
pointer-events: none;
.PipelineRegion-button {
position: absolute;
top: 0;
left: -60px;
width: 40px;
height: 40px;
border-radius: 50%;
color: inherit;
background: var(--tl-color-panel);
border: 1px solid currentColor;
display: flex;
align-items: center;
justify-content: center;
cursor: pointer;
.tl-container[data-state='select.idle'] & {
pointer-events: all;
}
svg {
color: hsl(260, 30%, 60%);
}
}
&::after {
content: '';
display: block;
width: 20px;
height: 0px;
border-bottom: 1px dashed currentColor;
position: absolute;
top: 20px;
left: -20px;
}
&:has(.PipelineRegion-button:hover) {
color: var(--tl-color-selected);
}
&.PipelineRegion_executing {
color: var(--tl-color-selected);
border-style: solid;
&::after {
border-bottom-style: solid;
}
.PipelineRegion-button {
border-width: 2px;
&::after {
content: '';
width: 8px;
height: 4px;
border-radius: 100%;
border: 2px solid var(--tl-color-selected);
background: var(--tl-color-panel);
position: absolute;
animation: PipelineRegion-button-spin 1s linear infinite;
}
}
}
}
@keyframes PipelineRegion-button-spin {
0% {
transform: rotate(0deg) translateY(-19px);
}
100% {
transform: rotate(360deg) translateY(-19px);
}
}
/* ---- Template picker ---- */
.TemplatePicker-popover {
width: 240px;
background: var(--tl-color-panel);
border-radius: var(--tl-radius-3);
box-shadow:
0 0 0 1px rgba(0, 0, 0, 0.06),
0 4px 12px rgba(0, 0, 0, 0.12);
overflow: hidden;
}
.TemplatePicker-header {
padding: var(--tl-space-3) var(--tl-space-4);
font-size: 12px;
font-weight: 600;
border-bottom: 1px solid var(--tl-color-divider);
}
.TemplatePicker-save {
display: flex;
gap: var(--tl-space-2);
padding: var(--tl-space-3) var(--tl-space-4);
border-bottom: 1px solid var(--tl-color-divider);
input {
flex: 1 1 auto;
border: 1px solid var(--tl-color-hint);
border-radius: var(--tl-radius-3);
padding: var(--tl-space-2) var(--tl-space-3);
font: inherit;
font-size: 11px;
outline: none;
&:focus {
border-color: var(--tl-color-selected);
}
}
}
.TemplatePicker-list {
max-height: 200px;
overflow-y: auto;
}
.TemplatePicker-empty {
padding: var(--tl-space-4);
font-size: 11px;
color: var(--tl-color-text-3);
text-align: center;
}
.TemplatePicker-item {
display: flex;
align-items: center;
.TemplatePicker-item-button {
flex: 1 1 auto;
}
}
/* ---- Image pipeline layout ---- */
.image-pipeline-layout {
display: grid;
grid-template-columns: 260px 1fr;
grid-template-rows: 1fr;
grid-template-areas:
'sidebar canvas'
'sidebar canvas';
}
.image-pipeline-sidebar {
grid-area: sidebar;
background: var(--tl-color-panel);
border-right: 1px solid var(--tl-color-divider);
overflow: hidden;
display: flex;
flex-direction: column;
}
/* ---- Sidebar node list ---- */
.ImagePipelineSidebar {
display: flex;
flex-direction: column;
height: 100%;
overflow: hidden;
}
.ImagePipelineSidebar-header {
padding: 0 12px;
height: 40px;
display: flex;
align-items: center;
font-weight: 500;
font-size: 12px;
flex: 0 0 auto;
border-bottom: 1px solid var(--tl-color-divider);
color: var(--tl-color-text-3);
}
.ImagePipelineSidebar-list {
flex: 1 1 auto;
overflow-y: auto;
}
.ImagePipelineSidebar-group {
margin: 8px 0px;
border-bottom: 1px solid var(--tl-color-divider);
&:last-child {
border-bottom: none;
}
}
.ImagePipelineSidebar-category {
padding: 8px 12px 0;
margin-bottom: 8px;
font-size: 11px;
font-weight: 500;
color: var(--tl-color-text-3);
}
.ImagePipelineSidebar-item {
position: relative;
display: flex;
align-items: center;
gap: 8px;
width: 100%;
height: 40px;
padding: 0 12px;
border: none;
background: transparent;
font-family: inherit;
font-size: 12px;
cursor: pointer;
color: var(--tl-color-text-1);
text-align: left;
touch-action: none;
&::after {
content: '';
display: block;
position: absolute;
inset: 4px;
border-radius: var(--tl-radius-2);
background: var(--tl-color-muted-2);
opacity: 0;
}
& > * {
position: relative;
z-index: 1;
}
}
@media (hover: hover) {
.ImagePipelineSidebar-item:hover::after {
opacity: 1;
}
}
.ImagePipelineSidebar-item:active::after {
opacity: 1;
}
.ImagePipelineSidebar-item-icon {
flex: 0 0 auto;
width: 18px;
height: 18px;
display: flex;
align-items: center;
justify-content: center;
color: var(--tl-color-text-1);
}
.ImagePipelineSidebar-item-title {
flex: 1 1 auto;
}
.image-pipeline-canvas {
grid-area: canvas;
background: var(--tl-color-background);
}

View File

@@ -0,0 +1,5 @@
import ReactDOM from 'react-dom/client'
import App from './App.tsx'
import './index.css'
ReactDOM.createRoot(document.getElementById('root') as HTMLElement).render(<App />)

View File

@@ -0,0 +1,367 @@
import classNames from 'classnames'
import { useCallback } from 'react'
import {
Circle2d,
Group2d,
HTMLContainer,
RecordProps,
Rectangle2d,
resizeBox,
ShapeUtil,
T,
TldrawUiButton,
TldrawUiButtonLabel,
TldrawUiDropdownMenuContent,
TldrawUiDropdownMenuGroup,
TldrawUiDropdownMenuItem,
TldrawUiDropdownMenuRoot,
TldrawUiDropdownMenuTrigger,
TLResizeInfo,
TLShape,
useEditor,
useValue,
} from 'tldraw'
import { PlayIcon } from '../components/icons/PlayIcon'
import { StopIcon } from '../components/icons/StopIcon'
import {
NODE_FOOTER_HEIGHT_PX,
NODE_HEADER_HEIGHT_PX,
NODE_ROW_BOTTOM_PADDING_PX,
NODE_ROW_HEADER_GAP_PX,
PORT_RADIUS_PX,
} from '../constants'
import { executionState, startExecution, stopExecution } from '../execution/executionState'
import { Port } from '../ports/Port'
import { getNodeOutputPortInfo, getNodePorts } from './nodePorts'
import { getNodeDefinition, getNodeHeightPx, getNodeWidthPx, NodeBody, NodeType } from './nodeTypes'
import { resizeNode } from './resizeNode'
import { NodeValue, STOP_EXECUTION } from './types/shared'
const NODE_TYPE = 'node'
declare module 'tldraw' {
export interface TLGlobalShapePropsMap {
[NODE_TYPE]: { node: NodeType; isOutOfDate: boolean }
}
}
export type NodeShape = TLShape<typeof NODE_TYPE>
export class NodeShapeUtil extends ShapeUtil<NodeShape> {
static override type = NODE_TYPE
static override props: RecordProps<NodeShape> = {
node: NodeType,
isOutOfDate: T.boolean,
}
getDefaultProps(): NodeShape['props'] {
return {
node: getNodeDefinition(this.editor, 'prompt').getDefault(),
isOutOfDate: false,
}
}
override canEdit(_shape: NodeShape) {
return false
}
override canResize(shape: NodeShape) {
return getNodeDefinition(this.editor, shape.props.node).canResizeNode
}
override hideResizeHandles(shape: NodeShape) {
return !this.canResize(shape)
}
override hideRotateHandle(_shape: NodeShape) {
return true
}
override hideSelectionBoundsBg(shape: NodeShape) {
return !this.canResize(shape)
}
override hideSelectionBoundsFg(shape: NodeShape) {
return !this.canResize(shape)
}
override isAspectRatioLocked(_shape: NodeShape) {
return false
}
override getBoundsSnapGeometry(_shape: NodeShape) {
return {
points: [{ x: 0, y: 0 }],
}
}
getGeometry(shape: NodeShape) {
const ports = getNodePorts(this.editor, shape)
const width = getNodeWidthPx(this.editor, shape)
const portGeometries = Object.values(ports).map(
(port) =>
new Circle2d({
x: port.x - PORT_RADIUS_PX,
y: port.y - PORT_RADIUS_PX,
radius: PORT_RADIUS_PX,
isFilled: true,
isLabel: true,
excludeFromShapeBounds: true,
})
)
const bodyGeometry = new Rectangle2d({
width,
height: getNodeHeightPx(this.editor, shape),
isFilled: true,
})
return new Group2d({
children: [bodyGeometry, ...portGeometries],
})
}
override onResize(shape: any, info: TLResizeInfo<any>) {
const definition = getNodeDefinition(this.editor, shape.props.node)
if (definition.canResizeNode) {
const node = shape.props.node as { w: number; h: number; type: string }
const prevW = getNodeWidthPx(this.editor, shape)
const prevH = getNodeHeightPx(this.editor, shape)
const newW = Math.max(200, Math.round(prevW * info.scaleX))
const newH = Math.max(120, Math.round(prevH * info.scaleY))
const bodyH =
newH -
NODE_HEADER_HEIGHT_PX -
NODE_ROW_HEADER_GAP_PX -
NODE_ROW_BOTTOM_PADDING_PX -
NODE_FOOTER_HEIGHT_PX
return {
...resizeNode(shape, info),
props: {
...shape.props,
node: {
...node,
w: newW,
h:
NODE_HEADER_HEIGHT_PX +
NODE_ROW_HEADER_GAP_PX +
Math.max(0, bodyH) +
NODE_ROW_BOTTOM_PADDING_PX +
NODE_FOOTER_HEIGHT_PX,
},
},
}
}
return resizeBox(shape, info)
}
component(shape: NodeShape) {
return <NodeShapeComponent shape={shape} />
}
getIndicatorPath(shape: NodeShape) {
const width = getNodeWidthPx(this.editor, shape)
const height = getNodeHeightPx(this.editor, shape)
const path = new Path2D()
path.rect(0, 0, width, height)
const ports = Object.values(getNodePorts(this.editor, shape))
for (const port of ports) {
path.moveTo(port.x + PORT_RADIUS_PX, port.y)
path.arc(port.x, port.y, PORT_RADIUS_PX, 0, Math.PI * 2)
}
return path
}
}
function NodeShapeComponent({ shape }: { shape: NodeShape }) {
const editor = useEditor()
const output = useValue(
'output',
() => getNodeOutputPortInfo(editor, shape.id)?.output ?? undefined,
[editor, shape.id]
)
const isExecuting = useValue(
'is executing',
() => executionState.get(editor).runningGraph?.getNodeStatus(shape.id) === 'executing',
[editor, shape.id]
)
const isGraphRunning = useValue(
'is graph running',
() => executionState.get(editor).runningGraph !== null,
[editor]
)
const nodeDefinition = getNodeDefinition(editor, shape.props.node)
return (
<HTMLContainer
className={classNames('NodeShape', {
NodeShape_executing: isExecuting,
NodeShape_capture: shape.props.node.type === 'capture',
})}
onContextMenu={(e) => {
const target = e.target as HTMLElement
const tag = target.tagName
if (tag === 'INPUT' || tag === 'TEXTAREA' || tag === 'SELECT') {
e.stopPropagation()
}
}}
>
<div className="NodeShape-heading">
<div className="NodeShape-icon">{nodeDefinition.icon}</div>
<div className="NodeShape-label">{nodeDefinition.heading ?? nodeDefinition.title}</div>
{output !== undefined && (
<>
<div className="NodeShape-output">
<NodeValue
value={
output.isOutOfDate
? STOP_EXECUTION
: output.multi
? output.value[0]
: output.value
}
/>
</div>
<Port shapeId={shape.id} portId="output" />
</>
)}
</div>
<NodeBody shape={shape} />
<div className="NodeShape-footer">
<button
className={classNames('NodeShape-footer-action', {
'NodeShape-footer-action_executing': isExecuting,
})}
onPointerDown={(e) => e.stopPropagation()}
onClick={() => {
if (isGraphRunning) {
stopExecution(editor)
} else {
startExecution(editor, new Set([shape.id]))
}
}}
>
{isExecuting ? <StopIcon /> : <PlayIcon />}
<span>{isExecuting ? 'Stop' : 'Play from here'}</span>
</button>
<NodeFooterMenu shape={shape} />
</div>
</HTMLContainer>
)
}
function NodeFooterMenu({ shape }: { shape: NodeShape }) {
const editor = useEditor()
const outputInfo = useValue('output info', () => getNodeOutputPortInfo(editor, shape.id), [
editor,
shape.id,
])
// Find any image output that has a valid URL
const imageUrl = Object.values(outputInfo).find(
(info) =>
info.dataType === 'image' && typeof info.value === 'string' && info.value && info.value !== ''
)?.value as string | undefined
const node = shape.props.node as Record<string, unknown>
const definition = getNodeDefinition(editor, shape.props.node)
const resultKeys = definition.resultKeys
const defaults = definition.getDefault() as Record<string, unknown>
const hasResult = resultKeys ? resultKeys.some((key) => node[key] !== defaults[key]) : false
const textOutput = Object.values(outputInfo).find(
(info) => info.dataType === 'text' && typeof info.value === 'string' && info.value !== ''
)?.value as string | undefined
const textResult =
textOutput ??
(typeof node.lastResultText === 'string' && node.lastResultText !== ''
? (node.lastResultText as string)
: null)
const handleDuplicate = useCallback(() => {
editor.markHistoryStoppingPoint('duplicate node')
editor.duplicateShapes([shape.id])
}, [editor, shape.id])
const handleDownloadImage = useCallback(async () => {
if (!imageUrl) return
const response = await fetch(imageUrl)
const blob = await response.blob()
const ext = blob.type.split('/')[1] ?? 'png'
const blobUrl = URL.createObjectURL(blob)
const a = document.createElement('a')
a.href = blobUrl
a.download = `image.${ext}`
document.body.appendChild(a)
a.click()
document.body.removeChild(a)
URL.revokeObjectURL(blobUrl)
}, [imageUrl])
const handleCopyText = useCallback(async () => {
if (!textResult) return
await navigator.clipboard.writeText(textResult)
}, [textResult])
const handleClearResult = useCallback(() => {
if (!resultKeys || resultKeys.length === 0) return
const updates: Record<string, unknown> = {}
for (const key of resultKeys) {
updates[key] = defaults[key]
}
editor.updateShape({
id: shape.id,
type: shape.type,
props: {
node: { ...(shape.props.node as any), ...updates },
isOutOfDate: true,
},
})
}, [editor, resultKeys, defaults, shape])
return (
<div className="NodeFooterMenu" onPointerDown={(e) => e.stopPropagation()}>
<TldrawUiDropdownMenuRoot id={`node-menu-${shape.id}`}>
<TldrawUiDropdownMenuTrigger>
<TldrawUiButton type="icon" title="More options" className="NodeFooterMenu-trigger">
<svg width="12" height="12" viewBox="0 0 12 12">
<circle cx="6" cy="2" r="1.2" fill="currentColor" />
<circle cx="6" cy="6" r="1.2" fill="currentColor" />
<circle cx="6" cy="10" r="1.2" fill="currentColor" />
</svg>
</TldrawUiButton>
</TldrawUiDropdownMenuTrigger>
<TldrawUiDropdownMenuContent side="top" align="end" sideOffset={4} alignOffset={0}>
<TldrawUiDropdownMenuGroup>
<TldrawUiDropdownMenuItem>
<TldrawUiButton type="menu" onClick={handleDuplicate}>
<TldrawUiButtonLabel>Duplicate</TldrawUiButtonLabel>
</TldrawUiButton>
</TldrawUiDropdownMenuItem>
{imageUrl && (
<TldrawUiDropdownMenuItem>
<TldrawUiButton type="menu" onClick={handleDownloadImage}>
<TldrawUiButtonLabel>Download image</TldrawUiButtonLabel>
</TldrawUiButton>
</TldrawUiDropdownMenuItem>
)}
{textResult && (
<TldrawUiDropdownMenuItem>
<TldrawUiButton type="menu" onClick={handleCopyText}>
<TldrawUiButtonLabel>Copy text</TldrawUiButtonLabel>
</TldrawUiButton>
</TldrawUiDropdownMenuItem>
)}
{hasResult && (
<TldrawUiDropdownMenuItem>
<TldrawUiButton type="menu" onClick={handleClearResult}>
<TldrawUiButtonLabel>Clear result</TldrawUiButtonLabel>
</TldrawUiButton>
</TldrawUiDropdownMenuItem>
)}
</TldrawUiDropdownMenuGroup>
</TldrawUiDropdownMenuContent>
</TldrawUiDropdownMenuRoot>
</div>
)
}

View File

@@ -0,0 +1,212 @@
/**
* This file contains functions for working with ports and connections on nodes.
*/
import { createComputedCache, Editor, TLShapeId } from 'tldraw'
import { getConnectionBindings } from '../connection/ConnectionBindingUtil'
import { PortDataType } from '../constants'
import { PortId } from '../ports/Port'
import { NodeShape } from './NodeShapeUtil'
import { getNodeOutputInfo, getNodeTypePorts } from './nodeTypes'
import {
areAnyInputsOutOfDate,
InfoValues,
isMultiInfoValue,
PipelineValue,
STOP_EXECUTION,
} from './types/shared'
/**
* Get the ports for a node. This is cached, because we only want to re-evaluate it when the
* underlying records change.
*/
export function getNodePorts(editor: Editor, shape: NodeShape | TLShapeId) {
return nodePortsCache.get(editor, typeof shape === 'string' ? shape : shape.id) ?? {}
}
const nodePortsCache = createComputedCache('ports', (editor: Editor, node: NodeShape) =>
getNodeTypePorts(editor, node)
)
/**
* A connection from one node to another.
*/
export interface NodePortConnection {
connectedShapeId: TLShapeId
connectionId: TLShapeId
terminal: 'start' | 'end'
ownPortId: PortId
connectedPortId: PortId
order: number
}
/**
* Get the connections for a node. This is cached.
*/
export function getNodePortConnections(
editor: Editor,
shape: NodeShape | TLShapeId
): NodePortConnection[] {
return nodePortConnectionsCache.get(editor, typeof shape === 'string' ? shape : shape.id) ?? []
}
const nodePortConnectionsCache = createComputedCache(
'port connections',
(editor: Editor, node: NodeShape) => {
const bindings = editor.getBindingsToShape(node.id, 'connection')
const connections: NodePortConnection[] = []
for (const binding of bindings) {
const oppositeTerminal = binding.props.terminal === 'start' ? 'end' : 'start'
const oppositeBinding = getConnectionBindings(editor, binding.fromId)[oppositeTerminal]
if (!oppositeBinding) continue
connections.push({
connectedShapeId: oppositeBinding.toId,
connectionId: binding.fromId,
terminal: binding.props.terminal,
ownPortId: binding.props.portId,
connectedPortId: oppositeBinding.props.portId,
order: binding.props.order ?? 0,
})
}
return connections
},
{
areRecordsEqual: (a, b) => a.id === b.id,
}
)
/**
* Get the values of the input ports for a node. This is cached.
*/
export function getNodeInputPortValues(editor: Editor, shape: NodeShape | TLShapeId): InfoValues {
return nodeInputPortValuesCache.get(editor, typeof shape === 'string' ? shape : shape.id) ?? {}
}
const nodeInputPortValuesCache = createComputedCache(
'node input port values',
(editor: Editor, node: NodeShape) => {
const connections = getNodePortConnections(editor, node)
const sorted = [...connections].sort((a, b) => a.order - b.order)
const ports = getNodePorts(editor, node)
const values: InfoValues = {}
for (const connection of sorted) {
if (!connection || connection.terminal !== 'end') continue
const connectedShapeOutputs = getNodeOutputPortInfo(editor, connection.connectedShapeId)
if (!connectedShapeOutputs) {
continue
}
const output = connectedShapeOutputs[connection.connectedPortId]
if (!output) continue
const port = ports[connection.ownPortId]
if (port?.multi) {
// Output ports are always single-valued; extract the scalar value.
const singleValue = isMultiInfoValue(output) ? output.value[0] : output.value
const existing = values[connection.ownPortId]
if (existing && isMultiInfoValue(existing)) {
existing.value.push(singleValue)
existing.isOutOfDate = existing.isOutOfDate || output.isOutOfDate
} else {
values[connection.ownPortId] = {
value: [singleValue],
isOutOfDate: output.isOutOfDate,
dataType: output.dataType,
multi: true,
}
}
} else {
values[connection.ownPortId] = output
}
}
return values
},
{
areRecordsEqual: (a, b) => a.props === b.props,
}
)
/**
* Get the values of the output ports for a node. This is cached.
*/
export function getNodeOutputPortInfo(editor: Editor, shape: NodeShape | TLShapeId): InfoValues {
return nodeOutputPortInfoCache.get(editor, typeof shape === 'string' ? shape : shape.id) ?? {}
}
function hasStopExecution(input: InfoValues[string]): boolean {
if (isMultiInfoValue(input)) {
return input.value.some((v) => v === STOP_EXECUTION)
}
return input.value === STOP_EXECUTION
}
const nodeOutputPortInfoCache = createComputedCache(
'node output port info',
(editor: Editor, node: NodeShape): InfoValues => {
const inputs = getNodeInputPortValues(editor, node)
if (Object.values(inputs).some(hasStopExecution)) {
const ports = getNodePorts(editor, node)
return Object.fromEntries(
Object.values(ports)
.filter((port) => port.terminal === 'start')
.map((port) => [
port.id,
{
value: STOP_EXECUTION as PipelineValue | STOP_EXECUTION,
isOutOfDate: areAnyInputsOutOfDate(inputs),
dataType: port.dataType,
},
])
)
}
return getNodeOutputInfo(editor, node, inputs as InfoValues)
},
{
areRecordsEqual: (a, b) => a.props === b.props,
}
)
/**
* Traverse the graph from a starting node, in a given direction, returning all the nodes that are
* connected to it (including the starting node).
*/
export function getAllConnectedNodes(
editor: Editor,
startingNode: TLShapeId | NodeShape,
direction?: 'start' | 'end'
) {
const toVisit = [typeof startingNode === 'string' ? startingNode : startingNode.id]
const found = new Set<TLShapeId>()
while (toVisit.length > 0) {
const nodeId = toVisit.shift()
if (!nodeId) continue
const node = editor.getShape(nodeId)
if (!node || !editor.isShapeOfType(node, 'node')) continue
if (found.has(node.id)) continue
found.add(node.id)
for (const connection of getNodePortConnections(editor, node)) {
if (direction && connection.terminal !== direction) continue
toVisit.push(connection.connectedShapeId)
}
}
return found
}
/**
* Get the data type of a port on a node. Returns the port's dataType.
*/
export function getPortDataType(
editor: Editor,
shapeId: TLShapeId,
portId: PortId
): PortDataType | null {
const ports = getNodePorts(editor, shapeId)
return ports[portId]?.dataType ?? null
}

View File

@@ -0,0 +1,138 @@
import { Editor, T, useEditor, WeakCache } from 'tldraw'
import {
NODE_FOOTER_HEIGHT_PX,
NODE_HEADER_HEIGHT_PX,
NODE_ROW_BOTTOM_PADDING_PX,
NODE_ROW_HEADER_GAP_PX,
} from '../constants'
import { PortId, ShapePort } from '../ports/Port'
import { NodeShape } from './NodeShapeUtil'
import { AdjustNodeDefinition } from './types/AdjustNode'
import { BlendNodeDefinition } from './types/BlendNode'
import { CaptureNodeDefinition } from './types/CaptureNode'
import { ControlNetNodeDefinition } from './types/ControlNetNode'
import { GenerateNodeDefinition } from './types/GenerateNode'
import { GenerateTextNodeDefinition } from './types/GenerateTextNode'
import { IPAdapterNodeDefinition } from './types/IPAdapterNode'
import { IteratorNodeDefinition } from './types/IteratorNode'
import { LoadImageNodeDefinition } from './types/LoadImageNode'
import { ModelNodeDefinition } from './types/ModelNode'
import { NumberNodeDefinition } from './types/NumberNode'
import { PreviewNodeDefinition } from './types/PreviewNode'
import { PromptConcatNodeDefinition } from './types/PromptConcatNode'
import { PromptNodeDefinition } from './types/PromptNode'
import { RouterNodeDefinition } from './types/RouterNode'
import {
ExecutionResult,
InfoValues,
NodeDefinition,
NodeDefinitionConstructor,
} from './types/shared'
import { StyleTransferNodeDefinition } from './types/StyleTransferNode'
import { UpscaleNodeDefinition } from './types/UpscaleNode'
/** All our node types */
export const NodeDefinitions = {
model: ModelNodeDefinition,
prompt: PromptNodeDefinition,
generate: GenerateNodeDefinition,
generate_text: GenerateTextNodeDefinition,
controlnet: ControlNetNodeDefinition,
load_image: LoadImageNodeDefinition,
preview: PreviewNodeDefinition,
blend: BlendNodeDefinition,
adjust: AdjustNodeDefinition,
upscale: UpscaleNodeDefinition,
ip_adapter: IPAdapterNodeDefinition,
style_transfer: StyleTransferNodeDefinition,
prompt_concat: PromptConcatNodeDefinition,
number: NumberNodeDefinition,
router: RouterNodeDefinition,
iterator: IteratorNodeDefinition,
capture: CaptureNodeDefinition,
} satisfies Record<string, NodeDefinitionConstructor<any>>
/**
* A union type of all our node types.
*/
export type NodeType = T.TypeOf<typeof NodeType>
export const NodeType = T.union(
'type',
Object.fromEntries(Object.values(NodeDefinitions).map((type) => [type.type, type.validator])) as {
[K in keyof typeof NodeDefinitions as (typeof NodeDefinitions)[K]['type']]: (typeof NodeDefinitions)[K]['validator']
}
)
const nodeDefinitions = new WeakCache<
Editor,
{ [K in keyof typeof NodeDefinitions]: InstanceType<(typeof NodeDefinitions)[K]> }
>()
export function getNodeDefinitions(editor: Editor) {
return nodeDefinitions.get(editor, () => {
return Object.fromEntries(
Object.values(NodeDefinitions).map((value) => [value.type, new value(editor)])
) as any
})
}
export function getNodeDefinition(
editor: Editor,
node: NodeType | NodeType['type']
): NodeDefinition<NodeType> {
return getNodeDefinitions(editor)[
typeof node === 'string' ? node : node.type
] as NodeDefinition<NodeType>
}
export function getNodeWidthPx(editor: Editor, shape: NodeShape): number {
return getNodeDefinition(editor, shape.props.node).getWidthPx(shape, shape.props.node)
}
export function getNodeBodyHeightPx(editor: Editor, shape: NodeShape): number {
return getNodeDefinition(editor, shape.props.node).getBodyHeightPx(shape, shape.props.node)
}
export function getNodeHeightPx(editor: Editor, shape: NodeShape): number {
return (
NODE_HEADER_HEIGHT_PX +
NODE_ROW_HEADER_GAP_PX +
getNodeBodyHeightPx(editor, shape) +
NODE_ROW_BOTTOM_PADDING_PX +
NODE_FOOTER_HEIGHT_PX
)
}
export function getNodeTypePorts(editor: Editor, shape: NodeShape): Record<string, ShapePort> {
return getNodeDefinition(editor, shape.props.node).getPorts(shape, shape.props.node)
}
export async function executeNode(
editor: Editor,
shape: NodeShape,
inputs: Record<string, string | number | null | (string | number | null)[]>
): Promise<ExecutionResult> {
return await getNodeDefinition(editor, shape.props.node).execute(shape, shape.props.node, inputs)
}
export function getNodeOutputInfo(
editor: Editor,
shape: NodeShape,
inputs: InfoValues
): InfoValues {
return getNodeDefinition(editor, shape.props.node).getOutputInfo(shape, shape.props.node, inputs)
}
export function onNodePortConnect(editor: Editor, shape: NodeShape, port: PortId) {
getNodeDefinition(editor, shape.props.node).onPortConnect?.(shape, shape.props.node, port)
}
export function onNodePortDisconnect(editor: Editor, shape: NodeShape, port: PortId) {
getNodeDefinition(editor, shape.props.node).onPortDisconnect?.(shape, shape.props.node, port)
}
export function NodeBody({ shape }: { shape: NodeShape }) {
const editor = useEditor()
const node = shape.props.node
const { Component } = getNodeDefinition(editor, node)
return <Component shape={shape} node={node} />
}

View File

@@ -0,0 +1,129 @@
import { Box, TLResizeHandle, TLResizeMode, Vec, VecModel } from 'tldraw'
import { NodeShape } from './NodeShapeUtil'
/** @public */
export interface ResizeBoxOptions {
minWidth?: number
maxWidth?: number
minHeight?: number
maxHeight?: number
}
/** @public */
export function resizeNode<T extends NodeShape & { props: { node: { w: number; h: number } } }>(
shape: T,
info: {
newPoint: VecModel
handle: TLResizeHandle
mode: TLResizeMode
scaleX: number
scaleY: number
initialBounds: Box
initialShape: T
},
opts = {} as ResizeBoxOptions
): T {
const { newPoint, handle, scaleX, scaleY } = info
const { minWidth = 1, maxWidth = Infinity, minHeight = 1, maxHeight = Infinity } = opts
let w = shape.props.node.w * scaleX
let h = shape.props.node.h * scaleY
const offset = new Vec(0, 0)
if (w > 0) {
if (w < minWidth) {
switch (handle) {
case 'top_left':
case 'left':
case 'bottom_left': {
offset.x = w - minWidth
break
}
case 'top':
case 'bottom': {
offset.x = (w - minWidth) / 2
break
}
default: {
offset.x = 0
}
}
w = minWidth
}
} else {
offset.x = w
w = -w
if (w < minWidth) {
switch (handle) {
case 'top_left':
case 'left':
case 'bottom_left': {
offset.x = -w
break
}
default: {
offset.x = -minWidth
}
}
w = minWidth
}
}
if (h > 0) {
if (h < minHeight) {
switch (handle) {
case 'top_left':
case 'top':
case 'top_right': {
offset.y = h - minHeight
break
}
case 'right':
case 'left': {
offset.y = (h - minHeight) / 2
break
}
default: {
offset.y = 0
}
}
h = minHeight
}
} else {
offset.y = h
h = -h
if (h < minHeight) {
switch (handle) {
case 'top_left':
case 'top':
case 'top_right': {
offset.y = -h
break
}
default: {
offset.y = -minHeight
}
}
h = minHeight
}
}
const { x, y } = offset.rot(shape.rotation).add(newPoint)
return {
...shape,
x,
y,
props: {
...shape.props,
node: {
...shape.props.node,
w: Math.min(maxWidth, w),
h: Math.min(maxHeight, h),
},
},
}
}

View File

@@ -0,0 +1,197 @@
import classNames from 'classnames'
import { T, useEditor, useValue } from 'tldraw'
import { AdjustIcon } from '../../components/icons/AdjustIcon'
import {
NODE_HEADER_HEIGHT_PX,
NODE_IMAGE_PREVIEW_HEIGHT_PX,
NODE_ROW_HEADER_GAP_PX,
NODE_ROW_HEIGHT_PX,
NODE_WIDTH_PX,
} from '../../constants'
import { Port, ShapePort } from '../../ports/Port'
import { getNodeInputPortValues } from '../nodePorts'
import { NodeShape } from '../NodeShapeUtil'
import {
areAnyInputsOutOfDate,
blobToDataUrl,
ExecutionResult,
InfoValues,
InputValues,
loadImage,
NodeComponentProps,
NodeDefinition,
NodeImage,
NodePlaceholder,
NodePortLabel,
NodeRow,
STOP_EXECUTION,
updateNode,
} from './shared'
export type AdjustNode = T.TypeOf<typeof AdjustNode>
export const AdjustNode = T.object({
type: T.literal('adjust'),
brightness: T.number,
contrast: T.number,
saturation: T.number,
lastResultUrl: T.string.nullable(),
})
async function adjustImage(
imageUrl: string,
brightness: number,
contrast: number,
saturation: number
): Promise<string> {
const img = await loadImage(imageUrl)
const w = img.naturalWidth
const h = img.naturalHeight
const canvas = new OffscreenCanvas(w, h)
const ctx = canvas.getContext('2d')!
// Map slider ranges (-50..50) to CSS filter multipliers
const b = 1 + brightness / 50 // 0.0 – 2.0, default 1.0
const c = 1 + contrast / 50 // 0.0 – 2.0, default 1.0
const s = 1 + saturation / 50 // 0.0 – 2.0, default 1.0
ctx.filter = `brightness(${b}) contrast(${c}) saturate(${s})`
ctx.drawImage(img, 0, 0, w, h)
const blob = await canvas.convertToBlob({ type: 'image/png' })
return blobToDataUrl(blob)
}
export class AdjustNodeDefinition extends NodeDefinition<AdjustNode> {
static type = 'adjust'
static validator = AdjustNode
title = 'Adjust'
heading = 'Adjust'
icon = <AdjustIcon />
category = 'process'
resultKeys = ['lastResultUrl'] as const
getDefault(): AdjustNode {
return {
type: 'adjust',
brightness: 0,
contrast: 0,
saturation: 0,
lastResultUrl: null,
}
}
getBodyHeightPx() {
return NODE_ROW_HEIGHT_PX * 4 + NODE_IMAGE_PREVIEW_HEIGHT_PX
}
getPorts(): Record<string, ShapePort> {
const baseY = NODE_HEADER_HEIGHT_PX + NODE_ROW_HEADER_GAP_PX
return {
image: {
id: 'image',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 0.5,
terminal: 'end',
dataType: 'image',
},
output: {
id: 'output',
x: NODE_WIDTH_PX,
y: NODE_HEADER_HEIGHT_PX / 2,
terminal: 'start',
dataType: 'image',
},
}
}
async execute(shape: NodeShape, node: AdjustNode, inputs: InputValues): Promise<ExecutionResult> {
const imageUrl = inputs.image as string | null
if (!imageUrl) {
updateNode<AdjustNode>(this.editor, shape, (n) => ({ ...n, lastResultUrl: null }))
return { output: null }
}
const result = await adjustImage(imageUrl, node.brightness, node.contrast, node.saturation)
updateNode<AdjustNode>(this.editor, shape, (n) => ({
...n,
lastResultUrl: result,
}))
return { output: result }
}
getOutputInfo(shape: NodeShape, node: AdjustNode, inputs: InfoValues): InfoValues {
return {
output: {
value: node.lastResultUrl,
isOutOfDate: areAnyInputsOutOfDate(inputs) || shape.props.isOutOfDate,
dataType: 'image',
},
}
}
Component = AdjustNodeComponent
}
function AdjustNodeComponent({ shape, node }: NodeComponentProps<AdjustNode>) {
const editor = useEditor()
const imageInput = useValue('image input', () => getNodeInputPortValues(editor, shape.id).image, [
editor,
shape.id,
])
const makeSlider = (
label: string,
field: 'brightness' | 'contrast' | 'saturation',
min: number,
max: number
) => (
<NodeRow className="NodeInputRow">
<span className="NodeInputRow-label">{label}</span>
<input
type="range"
min={min}
max={max}
value={node[field]}
onChange={(e) =>
updateNode<AdjustNode>(
editor,
shape,
(n) => ({
...n,
[field]: Number(e.target.value),
}),
false
)
}
onPointerDown={(e) => e.stopPropagation()}
/>
<span className="NodeRow-value">{node[field]}</span>
</NodeRow>
)
return (
<>
<NodeRow>
<Port shapeId={shape.id} portId="image" />
<NodePortLabel dataType="image">Image</NodePortLabel>
{imageInput ? (
<span className="NodeRow-connected-value">
{imageInput.isOutOfDate || imageInput.value === STOP_EXECUTION ? (
<NodePlaceholder />
) : (
'connected'
)}
</span>
) : (
<span className="NodeRow-disconnected">not connected</span>
)}
</NodeRow>
{makeSlider('Brightness', 'brightness', -50, 50)}
{makeSlider('Contrast', 'contrast', -50, 50)}
{makeSlider('Saturation', 'saturation', -50, 50)}
<div
className={classNames('NodeImagePreview', {
NodeImagePreview_loading: shape.props.isOutOfDate,
})}
>
{node.lastResultUrl ? (
<NodeImage src={node.lastResultUrl} alt="Adjusted" />
) : (
<div className="NodeImagePreview-empty">
<span>Connect an image</span>
</div>
)}
</div>
</>
)
}

View File

@@ -0,0 +1,246 @@
import classNames from 'classnames'
import { T, useEditor, useValue } from 'tldraw'
import { BlendIcon } from '../../components/icons/BlendIcon'
import {
NODE_HEADER_HEIGHT_PX,
NODE_IMAGE_PREVIEW_HEIGHT_PX,
NODE_ROW_HEADER_GAP_PX,
NODE_ROW_HEIGHT_PX,
NODE_WIDTH_PX,
} from '../../constants'
import { Port, ShapePort } from '../../ports/Port'
import { getNodeInputPortValues } from '../nodePorts'
import { NodeShape } from '../NodeShapeUtil'
import {
areAnyInputsOutOfDate,
blobToDataUrl,
ExecutionResult,
InfoValues,
InputValues,
loadImage,
NodeComponentProps,
NodeDefinition,
NodeImage,
NodePlaceholder,
NodePortLabel,
NodeRow,
STOP_EXECUTION,
updateNode,
} from './shared'
const BLEND_MODES = [
{ id: 'normal', label: 'Normal' },
{ id: 'multiply', label: 'Multiply' },
{ id: 'screen', label: 'Screen' },
{ id: 'overlay', label: 'Overlay' },
{ id: 'difference', label: 'Difference' },
] as const
export type BlendNode = T.TypeOf<typeof BlendNode>
export const BlendNode = T.object({
type: T.literal('blend'),
mode: T.string,
opacity: T.number,
lastResultUrl: T.string.nullable(),
})
const COMPOSITE_OPS: Record<string, GlobalCompositeOperation> = {
normal: 'source-over',
multiply: 'multiply',
screen: 'screen',
overlay: 'overlay',
difference: 'difference',
}
async function blendImages(
imageAUrl: string,
imageBUrl: string,
mode: string,
opacity: number
): Promise<string> {
const [imgA, imgB] = await Promise.all([loadImage(imageAUrl), loadImage(imageBUrl)])
const w = Math.max(imgA.naturalWidth, imgB.naturalWidth)
const h = Math.max(imgA.naturalHeight, imgB.naturalHeight)
const canvas = new OffscreenCanvas(w, h)
const ctx = canvas.getContext('2d')!
ctx.drawImage(imgA, 0, 0, w, h)
ctx.globalCompositeOperation = COMPOSITE_OPS[mode] ?? 'source-over'
ctx.globalAlpha = opacity / 100
ctx.drawImage(imgB, 0, 0, w, h)
const blob = await canvas.convertToBlob({ type: 'image/png' })
return blobToDataUrl(blob)
}
export class BlendNodeDefinition extends NodeDefinition<BlendNode> {
static type = 'blend'
static validator = BlendNode
title = 'Blend'
heading = 'Blend'
icon = <BlendIcon />
category = 'process'
resultKeys = ['lastResultUrl'] as const
getDefault(): BlendNode {
return {
type: 'blend',
mode: 'normal',
opacity: 50,
lastResultUrl: null,
}
}
getBodyHeightPx() {
return NODE_ROW_HEIGHT_PX * 4 + NODE_IMAGE_PREVIEW_HEIGHT_PX
}
getPorts(): Record<string, ShapePort> {
const baseY = NODE_HEADER_HEIGHT_PX + NODE_ROW_HEADER_GAP_PX
return {
imageA: {
id: 'imageA',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 0.5,
terminal: 'end',
dataType: 'image',
},
imageB: {
id: 'imageB',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 1.5,
terminal: 'end',
dataType: 'image',
},
output: {
id: 'output',
x: NODE_WIDTH_PX,
y: NODE_HEADER_HEIGHT_PX / 2,
terminal: 'start',
dataType: 'image',
},
}
}
async execute(shape: NodeShape, node: BlendNode, inputs: InputValues): Promise<ExecutionResult> {
const imageA = inputs.imageA as string | null
const imageB = inputs.imageB as string | null
let result: string | null = null
if (imageA && imageB) {
result = await blendImages(imageA, imageB, node.mode, node.opacity)
} else if (imageA) {
result = imageA
} else if (imageB) {
result = imageB
}
updateNode<BlendNode>(this.editor, shape, (n) => ({
...n,
lastResultUrl: result,
}))
return { output: result }
}
getOutputInfo(shape: NodeShape, node: BlendNode, inputs: InfoValues): InfoValues {
return {
output: {
value: node.lastResultUrl,
isOutOfDate: areAnyInputsOutOfDate(inputs) || shape.props.isOutOfDate,
dataType: 'image',
},
}
}
Component = BlendNodeComponent
}
function BlendNodeComponent({ shape, node }: NodeComponentProps<BlendNode>) {
const editor = useEditor()
const imageAInput = useValue(
'imageA input',
() => getNodeInputPortValues(editor, shape.id).imageA,
[editor, shape.id]
)
const imageBInput = useValue(
'imageB input',
() => getNodeInputPortValues(editor, shape.id).imageB,
[editor, shape.id]
)
return (
<>
<NodeRow>
<Port shapeId={shape.id} portId="imageA" />
<NodePortLabel dataType="image">Image A</NodePortLabel>
{imageAInput ? (
<span className="NodeRow-connected-value">
{imageAInput.isOutOfDate || imageAInput.value === STOP_EXECUTION ? (
<NodePlaceholder />
) : (
'connected'
)}
</span>
) : (
<span className="NodeRow-disconnected">not connected</span>
)}
</NodeRow>
<NodeRow>
<Port shapeId={shape.id} portId="imageB" />
<NodePortLabel dataType="image">Image B</NodePortLabel>
{imageBInput ? (
<span className="NodeRow-connected-value">
{imageBInput.isOutOfDate || imageBInput.value === STOP_EXECUTION ? (
<NodePlaceholder />
) : (
'connected'
)}
</span>
) : (
<span className="NodeRow-disconnected">not connected</span>
)}
</NodeRow>
<NodeRow>
<select
value={node.mode}
onChange={(e) =>
updateNode<BlendNode>(editor, shape, (n) => ({ ...n, mode: e.target.value }))
}
>
{BLEND_MODES.map((m) => (
<option key={m.id} value={m.id}>
{m.label}
</option>
))}
</select>
</NodeRow>
<NodeRow className="NodeInputRow">
<span className="NodeInputRow-label">Opacity</span>
<input
type="range"
min="0"
max="100"
value={node.opacity}
onChange={(e) =>
updateNode<BlendNode>(
editor,
shape,
(n) => ({
...n,
opacity: Number(e.target.value),
}),
false
)
}
onPointerDown={(e) => e.stopPropagation()}
/>
<span className="NodeRow-value">{node.opacity}%</span>
</NodeRow>
<div
className={classNames('NodeImagePreview', {
NodeImagePreview_loading: shape.props.isOutOfDate,
})}
>
{node.lastResultUrl ? (
<NodeImage src={node.lastResultUrl} alt="Blended" />
) : (
<div className="NodeImagePreview-empty">
<span>Connect images to blend</span>
</div>
)}
</div>
</>
)
}

View File

@@ -0,0 +1,144 @@
import { Box, T, useEditor, useValue } from 'tldraw'
import { CaptureIcon } from '../../components/icons/CaptureIcon'
import {
NODE_FOOTER_HEIGHT_PX,
NODE_HEADER_HEIGHT_PX,
NODE_ROW_BOTTOM_PADDING_PX,
NODE_ROW_HEADER_GAP_PX,
} from '../../constants'
import { ShapePort } from '../../ports/Port'
import { NodeShape } from '../NodeShapeUtil'
import {
ExecutionResult,
InfoValues,
NodeComponentProps,
NodeDefinition,
blobToDataUrl,
updateNode,
} from './shared'
export type CaptureNode = T.TypeOf<typeof CaptureNode>
export const CaptureNode = T.object({
type: T.literal('capture'),
w: T.number,
h: T.number,
lastCaptureUrl: T.string.nullable(),
})
export class CaptureNodeDefinition extends NodeDefinition<CaptureNode> {
static type = 'capture'
static validator = CaptureNode
title = 'Capture'
heading = 'Capture'
icon = <CaptureIcon />
category = 'input'
resultKeys = ['lastCaptureUrl'] as const
override canResizeNode = true
getDefault(): CaptureNode {
return {
type: 'capture',
w: 400,
h: 300,
lastCaptureUrl: null,
}
}
override getWidthPx(_shape: NodeShape, node: CaptureNode): number {
return node.w
}
getBodyHeightPx(_shape: NodeShape, node: CaptureNode): number {
return (
node.h -
NODE_HEADER_HEIGHT_PX -
NODE_ROW_HEADER_GAP_PX -
NODE_ROW_BOTTOM_PADDING_PX -
NODE_FOOTER_HEIGHT_PX
)
}
getPorts(_shape: NodeShape, node: CaptureNode): Record<string, ShapePort> {
return {
output: {
id: 'output',
x: node.w,
y: NODE_HEADER_HEIGHT_PX / 2,
terminal: 'start',
dataType: 'image',
},
}
}
async execute(shape: NodeShape, _node: CaptureNode): Promise<ExecutionResult> {
const bounds = this.editor.getShapePageBounds(shape.id)
if (!bounds) return { output: null }
const allIds = this.editor.getShapeIdsInsideBounds(bounds)
// Filter out the capture node itself, all node shapes, and connection shapes
const filtered = [...allIds].filter((id) => {
if (id === shape.id) return false
const s = this.editor.getShape(id)
if (!s) return false
return s.type !== 'node' && s.type !== 'connection'
})
if (filtered.length === 0) {
updateNode<CaptureNode>(this.editor, shape, (n) => ({
...n,
lastCaptureUrl: null,
}))
return { output: null }
}
const captureBox = new Box(bounds.x, bounds.y, bounds.w, bounds.h)
const result = await this.editor.toImage(filtered, {
bounds: captureBox,
padding: 0,
background: true,
format: 'png',
})
const url = await blobToDataUrl(result.blob)
updateNode<CaptureNode>(this.editor, shape, (n) => ({
...n,
lastCaptureUrl: url,
}))
return { output: url }
}
getOutputInfo(shape: NodeShape, node: CaptureNode): InfoValues {
return {
output: {
value: node.lastCaptureUrl,
isOutOfDate: shape.props.isOutOfDate,
dataType: 'image',
},
}
}
Component = CaptureNodeComponent
}
function CaptureNodeComponent(props: NodeComponentProps<CaptureNode>) {
const editor = useEditor()
const { shape } = props
const geometry = useValue('geometry', () => editor.getShapeGeometry(shape.id), [editor, shape.id])
return (
<div
className="CaptureNode-body"
style={{
width: geometry.bounds.w,
height:
geometry.bounds.h -
NODE_HEADER_HEIGHT_PX -
NODE_ROW_HEADER_GAP_PX -
NODE_ROW_BOTTOM_PADDING_PX -
NODE_FOOTER_HEIGHT_PX,
}}
/>
)
}

View File

@@ -0,0 +1,274 @@
import classNames from 'classnames'
import { T, useEditor, useValue } from 'tldraw'
import { apiGenerate } from '../../api/pipelineApi'
import { ControlNetIcon } from '../../components/icons/ControlNetIcon'
import {
NODE_HEADER_HEIGHT_PX,
NODE_IMAGE_PREVIEW_HEIGHT_PX,
NODE_ROW_HEADER_GAP_PX,
NODE_ROW_HEIGHT_PX,
NODE_WIDTH_PX,
} from '../../constants'
import { Port, ShapePort } from '../../ports/Port'
import { getNodeInputPortValues } from '../nodePorts'
import { NodeShape } from '../NodeShapeUtil'
import {
areAnyInputsOutOfDate,
ExecutionResult,
InfoValues,
InputValues,
NodeComponentProps,
NodeDefinition,
NodeImage,
NodePlaceholder,
NodePortLabel,
NodeRow,
STOP_EXECUTION,
updateNode,
} from './shared'
const CONTROL_MODES = [
{ id: 'canny', label: 'Canny edge' },
{ id: 'depth', label: 'Depth map' },
{ id: 'pose', label: 'Pose' },
{ id: 'segmentation', label: 'Segmentation' },
] as const
export type ControlNetNode = T.TypeOf<typeof ControlNetNode>
export const ControlNetNode = T.object({
type: T.literal('controlnet'),
mode: T.string,
strength: T.number,
steps: T.number,
lastResultUrl: T.string.nullable(),
})
export class ControlNetNodeDefinition extends NodeDefinition<ControlNetNode> {
static type = 'controlnet'
static validator = ControlNetNode
title = 'ControlNet'
heading = 'ControlNet'
icon = <ControlNetIcon />
category = 'process'
resultKeys = ['lastResultUrl'] as const
getDefault(): ControlNetNode {
return {
type: 'controlnet',
mode: 'canny',
strength: 75,
steps: 20,
lastResultUrl: null,
}
}
getBodyHeightPx() {
// model input + image input + prompt input + mode row + strength row + steps row + preview
return NODE_ROW_HEIGHT_PX * 6 + NODE_IMAGE_PREVIEW_HEIGHT_PX
}
getPorts(): Record<string, ShapePort> {
const baseY = NODE_HEADER_HEIGHT_PX + NODE_ROW_HEADER_GAP_PX
return {
model: {
id: 'model',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 0.5,
terminal: 'end',
dataType: 'model',
},
image: {
id: 'image',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 1.5,
terminal: 'end',
dataType: 'image',
},
prompt: {
id: 'prompt',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 2.5,
terminal: 'end',
dataType: 'text',
},
output: {
id: 'output',
x: NODE_WIDTH_PX,
y: NODE_HEADER_HEIGHT_PX / 2,
terminal: 'start',
dataType: 'image',
},
}
}
async execute(
shape: NodeShape,
node: ControlNetNode,
inputs: InputValues
): Promise<ExecutionResult> {
const model = (inputs.model as string) ?? 'stable-diffusion:sdxl'
const prompt = (inputs.prompt as string) ?? ''
const referenceImageUrl = (inputs.image as string) ?? undefined
const result = await apiGenerate({
model,
prompt,
steps: node.steps,
cfgScale: 7,
controlNetMode: node.mode,
controlNetStrength: node.strength,
referenceImageUrl,
})
updateNode<ControlNetNode>(this.editor, shape, (n) => ({
...n,
lastResultUrl: result.imageUrl,
}))
return { output: result.imageUrl }
}
getOutputInfo(shape: NodeShape, node: ControlNetNode, inputs: InfoValues): InfoValues {
return {
output: {
value: node.lastResultUrl,
isOutOfDate: areAnyInputsOutOfDate(inputs) || shape.props.isOutOfDate,
dataType: 'image',
},
}
}
Component = ControlNetNodeComponent
}
function ControlNetNodeComponent({ shape, node }: NodeComponentProps<ControlNetNode>) {
const editor = useEditor()
const modelInput = useValue('model input', () => getNodeInputPortValues(editor, shape.id).model, [
editor,
shape.id,
])
const imageInput = useValue('image input', () => getNodeInputPortValues(editor, shape.id).image, [
editor,
shape.id,
])
const promptInput = useValue(
'prompt input',
() => getNodeInputPortValues(editor, shape.id).prompt,
[editor, shape.id]
)
return (
<>
<NodeRow>
<Port shapeId={shape.id} portId="model" />
<NodePortLabel dataType="model">Model</NodePortLabel>
{modelInput ? (
<span className="NodeRow-connected-value">
{modelInput.isOutOfDate || modelInput.value === STOP_EXECUTION ? (
<NodePlaceholder />
) : (
String(modelInput.value)
)}
</span>
) : (
<span className="NodeRow-disconnected">not connected</span>
)}
</NodeRow>
<NodeRow>
<Port shapeId={shape.id} portId="image" />
<NodePortLabel dataType="image">Reference</NodePortLabel>
{imageInput ? (
<span className="NodeRow-connected-value">
{imageInput.isOutOfDate || imageInput.value === STOP_EXECUTION ? (
<NodePlaceholder />
) : (
'connected'
)}
</span>
) : (
<span className="NodeRow-disconnected">not connected</span>
)}
</NodeRow>
<NodeRow>
<Port shapeId={shape.id} portId="prompt" />
<NodePortLabel dataType="text">Prompt</NodePortLabel>
{promptInput ? (
<span className="NodeRow-connected-value">
{promptInput.isOutOfDate || promptInput.value === STOP_EXECUTION ? (
<NodePlaceholder />
) : typeof promptInput.value === 'string' ? (
promptInput.value.slice(0, 15) + (promptInput.value.length > 15 ? '...' : '')
) : (
'connected'
)}
</span>
) : (
<span className="NodeRow-disconnected">not connected</span>
)}
</NodeRow>
<NodeRow>
<span className="NodeInputRow-label">Mode</span>
<select
value={node.mode}
onChange={(e) =>
updateNode<ControlNetNode>(editor, shape, (n) => ({
...n,
mode: e.target.value,
}))
}
>
{CONTROL_MODES.map((m) => (
<option key={m.id} value={m.id}>
{m.label}
</option>
))}
</select>
</NodeRow>
<NodeRow className="NodeInputRow">
<span className="NodeInputRow-label">Strength</span>
<input
type="range"
min="0"
max="100"
value={node.strength}
onChange={(e) =>
updateNode<ControlNetNode>(
editor,
shape,
(n) => ({ ...n, strength: Number(e.target.value) }),
false
)
}
onPointerDown={(e) => e.stopPropagation()}
/>
<span className="NodeRow-value">{node.strength}%</span>
</NodeRow>
<NodeRow className="NodeInputRow">
<span className="NodeInputRow-label">Steps</span>
<input
type="range"
min="1"
max="50"
value={node.steps}
onChange={(e) =>
updateNode<ControlNetNode>(
editor,
shape,
(n) => ({ ...n, steps: Number(e.target.value) }),
false
)
}
onPointerDown={(e) => e.stopPropagation()}
/>
<span className="NodeRow-value">{node.steps}</span>
</NodeRow>
<div
className={classNames('NodeImagePreview', {
NodeImagePreview_loading: shape.props.isOutOfDate,
})}
>
{node.lastResultUrl ? (
<NodeImage src={node.lastResultUrl} alt="ControlNet result" />
) : (
<div className="NodeImagePreview-empty">
<span>Connect model + reference image</span>
</div>
)}
</div>
</>
)
}

View File

@@ -0,0 +1,314 @@
import classNames from 'classnames'
import { T, useEditor, useValue } from 'tldraw'
import { apiGenerate } from '../../api/pipelineApi'
import { GenerateIcon } from '../../components/icons/GenerateIcon'
import {
NODE_HEADER_HEIGHT_PX,
NODE_IMAGE_PREVIEW_HEIGHT_PX,
NODE_ROW_HEADER_GAP_PX,
NODE_ROW_HEIGHT_PX,
NODE_WIDTH_PX,
} from '../../constants'
import { Port, ShapePort } from '../../ports/Port'
import { getNodeInputPortValues } from '../nodePorts'
import { NodeShape } from '../NodeShapeUtil'
import {
areAnyInputsOutOfDate,
ExecutionResult,
InfoValues,
InputValues,
isMultiInfoValue,
NodeComponentProps,
NodeDefinition,
NodeImage,
NodePlaceholder,
NodePortLabel,
NodeRow,
STOP_EXECUTION,
updateNode,
} from './shared'
export type GenerateNode = T.TypeOf<typeof GenerateNode>
export const GenerateNode = T.object({
type: T.literal('generate'),
steps: T.number,
cfgScale: T.number,
seed: T.number,
lastResultUrl: T.string.nullable(),
})
export class GenerateNodeDefinition extends NodeDefinition<GenerateNode> {
static type = 'generate'
static validator = GenerateNode
title = 'Generate'
heading = 'Generate'
icon = <GenerateIcon />
category = 'process'
resultKeys = ['lastResultUrl'] as const
getDefault(): GenerateNode {
return {
type: 'generate',
steps: 20,
cfgScale: 7,
seed: Math.floor(Math.random() * 99999),
lastResultUrl: null,
}
}
getBodyHeightPx() {
// 4 port rows + image preview + 3 parameter rows
return NODE_ROW_HEIGHT_PX * 7 + NODE_IMAGE_PREVIEW_HEIGHT_PX
}
getPorts(_shape: NodeShape, _node: GenerateNode): Record<string, ShapePort> {
const baseY = NODE_HEADER_HEIGHT_PX + NODE_ROW_HEADER_GAP_PX
return {
model: {
id: 'model',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 0.5,
terminal: 'end',
dataType: 'model',
},
prompt: {
id: 'prompt',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 1.5,
terminal: 'end',
dataType: 'text',
multi: true,
},
negative: {
id: 'negative',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 2.5,
terminal: 'end',
dataType: 'text',
},
image: {
id: 'image',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 3.5,
terminal: 'end',
dataType: 'image',
},
output: {
id: 'output',
x: NODE_WIDTH_PX,
y: NODE_HEADER_HEIGHT_PX / 2,
terminal: 'start',
dataType: 'image',
},
}
}
async execute(
shape: NodeShape,
node: GenerateNode,
inputs: InputValues
): Promise<ExecutionResult> {
const model = (inputs.model as string) ?? 'flux:flux-dev'
const rawPrompt = inputs.prompt
const promptValues = Array.isArray(rawPrompt) ? rawPrompt : [rawPrompt ?? 'default']
const prompt =
promptValues
.filter((v) => v != null)
.map(String)
.join(', ') || 'default'
const negativePrompt = inputs.negative as string | undefined
const referenceImageUrl = (inputs.image as string) ?? undefined
const result = await apiGenerate({
model,
prompt,
negativePrompt: negativePrompt ?? undefined,
steps: node.steps,
cfgScale: node.cfgScale,
seed: node.seed,
referenceImageUrl,
})
updateNode<GenerateNode>(this.editor, shape, (n) => ({
...n,
lastResultUrl: result.imageUrl,
}))
return { output: result.imageUrl }
}
getOutputInfo(shape: NodeShape, node: GenerateNode, inputs: InfoValues): InfoValues {
return {
output: {
value: node.lastResultUrl,
isOutOfDate: areAnyInputsOutOfDate(inputs) || shape.props.isOutOfDate,
dataType: 'image',
},
}
}
Component = GenerateNodeComponent
}
function GenerateNodeComponent({ shape, node }: NodeComponentProps<GenerateNode>) {
const editor = useEditor()
const modelInput = useValue('model input', () => getNodeInputPortValues(editor, shape.id).model, [
editor,
shape.id,
])
const promptInput = useValue(
'prompt input',
() => getNodeInputPortValues(editor, shape.id).prompt,
[editor, shape.id]
)
const negativeInput = useValue(
'negative input',
() => getNodeInputPortValues(editor, shape.id).negative,
[editor, shape.id]
)
const imageInput = useValue('image input', () => getNodeInputPortValues(editor, shape.id).image, [
editor,
shape.id,
])
return (
<>
<NodeRow>
<Port shapeId={shape.id} portId="model" />
<NodePortLabel dataType="model">Model</NodePortLabel>
{modelInput ? (
<span className="NodeRow-connected-value">
{modelInput.isOutOfDate ? <NodePlaceholder /> : String(modelInput.value)}
</span>
) : (
<span className="NodeRow-disconnected">not connected</span>
)}
</NodeRow>
<NodeRow>
<Port shapeId={shape.id} portId="prompt" />
<NodePortLabel dataType="text">Prompt</NodePortLabel>
{promptInput ? (
<span className="NodeRow-connected-value">
{promptInput.isOutOfDate ? (
<NodePlaceholder />
) : (
(() => {
const display = isMultiInfoValue(promptInput)
? promptInput.value.filter((v): v is string => typeof v === 'string').join(', ')
: String(promptInput.value ?? '')
return (
<span title={display}>
{display.slice(0, 20)}
{display.length > 20 ? '...' : ''}
</span>
)
})()
)}
</span>
) : (
<span className="NodeRow-disconnected">not connected</span>
)}
</NodeRow>
<NodeRow>
<Port shapeId={shape.id} portId="negative" />
<NodePortLabel dataType="text">Negative</NodePortLabel>
{negativeInput ? (
<span className="NodeRow-connected-value">
{negativeInput.isOutOfDate || negativeInput.value === STOP_EXECUTION ? (
<NodePlaceholder />
) : (
<span title={String(negativeInput.value)}>
{String(negativeInput.value ?? '').slice(0, 20)}
</span>
)}
</span>
) : (
<span className="NodeRow-disconnected">optional</span>
)}
</NodeRow>
<NodeRow>
<Port shapeId={shape.id} portId="image" />
<NodePortLabel dataType="image">Ref image</NodePortLabel>
{imageInput ? (
<span className="NodeRow-connected-value">
{imageInput.isOutOfDate ? <NodePlaceholder /> : 'connected'}
</span>
) : (
<span className="NodeRow-disconnected">optional</span>
)}
</NodeRow>
<div
className={classNames('NodeImagePreview', {
NodeImagePreview_loading: shape.props.isOutOfDate,
})}
>
{node.lastResultUrl ? (
<NodeImage src={node.lastResultUrl} alt="Generated" />
) : (
<div className="NodeImagePreview-empty">
<span>Run pipeline to generate</span>
</div>
)}
</div>
<NodeRow className="NodeInputRow">
<span className="NodeInputRow-label">Steps</span>
<input
type="range"
min="1"
max="100"
value={node.steps}
onChange={(e) =>
updateNode<GenerateNode>(
editor,
shape,
(n) => ({
...n,
steps: Number(e.target.value),
}),
false
)
}
onPointerDown={(e) => e.stopPropagation()}
/>
<span className="NodeRow-value">{node.steps}</span>
</NodeRow>
<NodeRow className="NodeInputRow">
<span className="NodeInputRow-label">CFG</span>
<input
type="range"
min="1"
max="30"
step="0.5"
value={node.cfgScale}
onChange={(e) =>
updateNode<GenerateNode>(
editor,
shape,
(n) => ({
...n,
cfgScale: Number(e.target.value),
}),
false
)
}
onPointerDown={(e) => e.stopPropagation()}
/>
<span className="NodeRow-value">{node.cfgScale}</span>
</NodeRow>
<NodeRow className="NodeInputRow">
<span className="NodeInputRow-label">Seed</span>
<input
type="text"
inputMode="numeric"
value={node.seed}
onChange={(e) => {
const v = parseInt(e.target.value, 10)
if (!isNaN(v)) {
updateNode<GenerateNode>(editor, shape, (n) => ({
...n,
seed: Math.max(0, v),
}))
}
}}
onPointerDown={(e) => e.stopPropagation()}
onFocus={() => editor.setSelectedShapes([shape.id])}
/>
</NodeRow>
</>
)
}

View File

@@ -0,0 +1,182 @@
import classNames from 'classnames'
import { T, useEditor, useValue } from 'tldraw'
import { apiGenerateText } from '../../api/pipelineApi'
import { GenerateTextIcon } from '../../components/icons/GenerateTextIcon'
import {
NODE_HEADER_HEIGHT_PX,
NODE_ROW_HEADER_GAP_PX,
NODE_ROW_HEIGHT_PX,
NODE_WIDTH_PX,
} from '../../constants'
import { Port, ShapePort } from '../../ports/Port'
import { getNodeInputPortValues } from '../nodePorts'
import { NodeShape } from '../NodeShapeUtil'
import {
areAnyInputsOutOfDate,
coerceToText,
ExecutionResult,
getInput,
getInputText,
InfoValues,
InputValues,
NodeComponentProps,
NodeDefinition,
NodePlaceholder,
NodePortLabel,
NodeRow,
updateNode,
} from './shared'
export type GenerateTextNode = T.TypeOf<typeof GenerateTextNode>
export const GenerateTextNode = T.object({
type: T.literal('generate_text'),
lastResultText: T.string.nullable(),
})
const DEFAULT_PROMPT = 'Describe this image in detail.'
export class GenerateTextNodeDefinition extends NodeDefinition<GenerateTextNode> {
static type = 'generate_text'
static validator = GenerateTextNode
title = 'Generate text'
heading = 'Generate text'
icon = <GenerateTextIcon />
category = 'process'
resultKeys = ['lastResultText'] as const
getDefault(): GenerateTextNode {
return {
type: 'generate_text',
lastResultText: null,
}
}
getBodyHeightPx() {
// input row (44) + prompt row (44) + result area (88 + 8 margin)
return NODE_ROW_HEIGHT_PX * 2 + 96
}
getPorts(_shape: NodeShape, _node: GenerateTextNode): Record<string, ShapePort> {
const baseY = NODE_HEADER_HEIGHT_PX + NODE_ROW_HEADER_GAP_PX
return {
input: {
id: 'input',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 0.5,
terminal: 'end',
dataType: 'any',
},
prompt: {
id: 'prompt',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 1.5,
terminal: 'end',
dataType: 'text',
},
output: {
id: 'output',
x: NODE_WIDTH_PX,
y: NODE_HEADER_HEIGHT_PX / 2,
terminal: 'start',
dataType: 'text',
},
}
}
async execute(
shape: NodeShape,
_node: GenerateTextNode,
inputs: InputValues
): Promise<ExecutionResult> {
const input = coerceToText(getInput(inputs, 'input')) || undefined
const prompt = getInputText(inputs, 'prompt', DEFAULT_PROMPT)
const result = await apiGenerateText({
input,
prompt,
})
updateNode<GenerateTextNode>(this.editor, shape, (n) => ({
...n,
lastResultText: result.text,
}))
return { output: result.text }
}
getOutputInfo(shape: NodeShape, node: GenerateTextNode, inputs: InfoValues): InfoValues {
return {
output: {
value: node.lastResultText,
isOutOfDate: areAnyInputsOutOfDate(inputs) || shape.props.isOutOfDate,
dataType: 'text',
},
}
}
Component = GenerateTextNodeComponent
}
function GenerateTextNodeComponent({ shape, node }: NodeComponentProps<GenerateTextNode>) {
const editor = useEditor()
const inputPort = useValue('input port', () => getNodeInputPortValues(editor, shape.id).input, [
editor,
shape.id,
])
const promptPort = useValue(
'prompt port',
() => getNodeInputPortValues(editor, shape.id).prompt,
[editor, shape.id]
)
return (
<>
<NodeRow>
<Port shapeId={shape.id} portId="input" />
<NodePortLabel dataType="any">Input</NodePortLabel>
{inputPort ? (
<span className="NodeRow-connected-value">
{inputPort.isOutOfDate ? (
<NodePlaceholder />
) : (
<span title={String(inputPort.value)}>
{String(inputPort.value ?? '').slice(0, 20)}
{String(inputPort.value ?? '').length > 20 ? '...' : ''}
</span>
)}
</span>
) : (
<span className="NodeRow-disconnected">not connected</span>
)}
</NodeRow>
<NodeRow>
<Port shapeId={shape.id} portId="prompt" />
<NodePortLabel dataType="text">Prompt</NodePortLabel>
{promptPort ? (
<span className="NodeRow-connected-value">
{promptPort.isOutOfDate ? (
<NodePlaceholder />
) : (
<span title={String(promptPort.value)}>
{String(promptPort.value ?? '').slice(0, 20)}
{String(promptPort.value ?? '').length > 20 ? '...' : ''}
</span>
)}
</span>
) : (
<span className="NodeRow-disconnected">optional</span>
)}
</NodeRow>
<div
className={classNames('GenerateTextNode-result', {
'GenerateTextNode-result_loading': shape.props.isOutOfDate,
})}
onPointerDown={(e) => e.stopPropagation()}
onWheel={(e) => e.stopPropagation()}
>
{node.lastResultText ? (
<div className="GenerateTextNode-result-text">{node.lastResultText}</div>
) : (
<div className="GenerateTextNode-result-empty">
<span>Run to generate text</span>
</div>
)}
</div>
</>
)
}

View File

@@ -0,0 +1,218 @@
import classNames from 'classnames'
import { T, useEditor, useValue } from 'tldraw'
import { apiIPAdapter } from '../../api/pipelineApi'
import { IPAdapterIcon } from '../../components/icons/IPAdapterIcon'
import {
NODE_HEADER_HEIGHT_PX,
NODE_IMAGE_PREVIEW_HEIGHT_PX,
NODE_ROW_HEADER_GAP_PX,
NODE_ROW_HEIGHT_PX,
NODE_WIDTH_PX,
} from '../../constants'
import { Port, ShapePort } from '../../ports/Port'
import { getNodeInputPortValues } from '../nodePorts'
import { NodeShape } from '../NodeShapeUtil'
import {
areAnyInputsOutOfDate,
ExecutionResult,
InfoValues,
InputValues,
NodeComponentProps,
NodeDefinition,
NodeImage,
NodePlaceholder,
NodePortLabel,
NodeRow,
STOP_EXECUTION,
updateNode,
} from './shared'
export type IPAdapterNode = T.TypeOf<typeof IPAdapterNode>
export const IPAdapterNode = T.object({
type: T.literal('ip_adapter'),
scale: T.number,
steps: T.number,
lastResultUrl: T.string.nullable(),
})
export class IPAdapterNodeDefinition extends NodeDefinition<IPAdapterNode> {
static type = 'ip_adapter'
static validator = IPAdapterNode
title = 'IP-Adapter'
heading = 'IP-Adapter'
hidden = true as const
icon = <IPAdapterIcon />
category = 'process'
resultKeys = ['lastResultUrl'] as const
getDefault(): IPAdapterNode {
return {
type: 'ip_adapter',
scale: 60,
steps: 30,
lastResultUrl: null,
}
}
getBodyHeightPx() {
// image input + prompt input + scale row + steps row + preview
return NODE_ROW_HEIGHT_PX * 4 + NODE_IMAGE_PREVIEW_HEIGHT_PX
}
getPorts(): Record<string, ShapePort> {
const baseY = NODE_HEADER_HEIGHT_PX + NODE_ROW_HEADER_GAP_PX
return {
image: {
id: 'image',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 0.5,
terminal: 'end',
dataType: 'image',
},
prompt: {
id: 'prompt',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 1.5,
terminal: 'end',
dataType: 'text',
},
output: {
id: 'output',
x: NODE_WIDTH_PX,
y: NODE_HEADER_HEIGHT_PX / 2,
terminal: 'start',
dataType: 'image',
},
}
}
async execute(
shape: NodeShape,
node: IPAdapterNode,
inputs: InputValues
): Promise<ExecutionResult> {
const imageUrl = (inputs.image as string) ?? ''
const prompt = (inputs.prompt as string) ?? ''
const result = await apiIPAdapter({
imageUrl,
prompt,
scale: node.scale / 100,
steps: node.steps,
})
updateNode<IPAdapterNode>(this.editor, shape, (n) => ({
...n,
lastResultUrl: result.imageUrl,
}))
return { output: result.imageUrl }
}
getOutputInfo(shape: NodeShape, node: IPAdapterNode, inputs: InfoValues): InfoValues {
return {
output: {
value: node.lastResultUrl,
isOutOfDate: areAnyInputsOutOfDate(inputs) || shape.props.isOutOfDate,
dataType: 'image',
},
}
}
Component = IPAdapterNodeComponent
}
function IPAdapterNodeComponent({ shape, node }: NodeComponentProps<IPAdapterNode>) {
const editor = useEditor()
const imageInput = useValue('image input', () => getNodeInputPortValues(editor, shape.id).image, [
editor,
shape.id,
])
const promptInput = useValue(
'prompt input',
() => getNodeInputPortValues(editor, shape.id).prompt,
[editor, shape.id]
)
return (
<>
<NodeRow>
<Port shapeId={shape.id} portId="image" />
<NodePortLabel dataType="image">Reference</NodePortLabel>
{imageInput ? (
<span className="NodeRow-connected-value">
{imageInput.isOutOfDate || imageInput.value === STOP_EXECUTION ? (
<NodePlaceholder />
) : (
'connected'
)}
</span>
) : (
<span className="NodeRow-disconnected">not connected</span>
)}
</NodeRow>
<NodeRow>
<Port shapeId={shape.id} portId="prompt" />
<NodePortLabel dataType="text">Prompt</NodePortLabel>
{promptInput ? (
<span className="NodeRow-connected-value">
{promptInput.isOutOfDate || promptInput.value === STOP_EXECUTION ? (
<NodePlaceholder />
) : typeof promptInput.value === 'string' ? (
promptInput.value.slice(0, 15) + (promptInput.value.length > 15 ? '...' : '')
) : (
'connected'
)}
</span>
) : (
<span className="NodeRow-disconnected">not connected</span>
)}
</NodeRow>
<NodeRow className="NodeInputRow">
<span className="NodeInputRow-label">Scale</span>
<input
type="range"
min="0"
max="100"
value={node.scale}
onChange={(e) =>
updateNode<IPAdapterNode>(
editor,
shape,
(n) => ({ ...n, scale: Number(e.target.value) }),
false
)
}
onPointerDown={(e) => e.stopPropagation()}
/>
<span className="NodeRow-value">{node.scale}%</span>
</NodeRow>
<NodeRow className="NodeInputRow">
<span className="NodeInputRow-label">Steps</span>
<input
type="range"
min="1"
max="50"
value={node.steps}
onChange={(e) =>
updateNode<IPAdapterNode>(
editor,
shape,
(n) => ({ ...n, steps: Number(e.target.value) }),
false
)
}
onPointerDown={(e) => e.stopPropagation()}
/>
<span className="NodeRow-value">{node.steps}</span>
</NodeRow>
<div
className={classNames('NodeImagePreview', {
NodeImagePreview_loading: shape.props.isOutOfDate,
})}
>
{node.lastResultUrl ? (
<NodeImage src={node.lastResultUrl} alt="IP-Adapter result" />
) : (
<div className="NodeImagePreview-empty">
<span>Connect a reference image</span>
</div>
)}
</div>
</>
)
}

View File

@@ -0,0 +1,235 @@
import classNames from 'classnames'
import { T, useEditor, useValue } from 'tldraw'
import { apiGenerate } from '../../api/pipelineApi'
import { IteratorIcon } from '../../components/icons/IteratorIcon'
import {
NODE_HEADER_HEIGHT_PX,
NODE_IMAGE_PREVIEW_HEIGHT_PX,
NODE_ROW_HEADER_GAP_PX,
NODE_ROW_HEIGHT_PX,
NODE_WIDTH_PX,
} from '../../constants'
import { Port, ShapePort } from '../../ports/Port'
import { getNodeInputPortValues } from '../nodePorts'
import { NodeShape } from '../NodeShapeUtil'
import {
areAnyInputsOutOfDate,
ExecutionResult,
getInputText,
InfoValues,
InputValues,
NodeComponentProps,
NodeDefinition,
NodeImage,
NodePortLabel,
NodeRow,
updateNode,
} from './shared'
export type IteratorNode = T.TypeOf<typeof IteratorNode>
export const IteratorNode = T.object({
type: T.literal('iterator'),
items: T.string,
completedCount: T.number,
totalCount: T.number,
lastResultUrl: T.string.nullable(),
})
export class IteratorNodeDefinition extends NodeDefinition<IteratorNode> {
static type = 'iterator'
static validator = IteratorNode
title = 'Iterator'
heading = 'Iterator'
hidden = true as const
icon = <IteratorIcon />
category = 'utility'
resultKeys = ['lastResultUrl', 'completedCount', 'totalCount'] as const
getDefault(): IteratorNode {
return {
type: 'iterator',
items: 'cat\ndog\nbird',
completedCount: 0,
totalCount: 0,
lastResultUrl: null,
}
}
getBodyHeightPx() {
// template input row + items textarea (2 rows) + progress row + preview
return NODE_ROW_HEIGHT_PX * 4 + NODE_IMAGE_PREVIEW_HEIGHT_PX
}
getPorts(): Record<string, ShapePort> {
const baseY = NODE_HEADER_HEIGHT_PX + NODE_ROW_HEADER_GAP_PX
return {
template: {
id: 'template',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 0.5,
terminal: 'end',
dataType: 'any',
},
output: {
id: 'output',
x: NODE_WIDTH_PX,
y: NODE_HEADER_HEIGHT_PX / 2,
terminal: 'start',
dataType: 'image',
},
current_item: {
id: 'current_item',
x: NODE_WIDTH_PX,
y: baseY + NODE_ROW_HEIGHT_PX * 0.5,
terminal: 'start',
dataType: 'text',
},
}
}
async execute(
shape: NodeShape,
node: IteratorNode,
inputs: InputValues
): Promise<ExecutionResult> {
const items = node.items
.split('\n')
.map((s) => s.trim())
.filter((s) => s.length > 0)
if (items.length === 0) {
updateNode<IteratorNode>(this.editor, shape, (n) => ({
...n,
completedCount: 0,
totalCount: 0,
lastResultUrl: null,
}))
return { output: null, current_item: null }
}
updateNode<IteratorNode>(this.editor, shape, (n) => ({
...n,
completedCount: 0,
totalCount: items.length,
}))
const template = getInputText(inputs, 'template')
let lastResult: string | null = null
for (let i = 0; i < items.length; i++) {
const prompt = template ? `${template}, ${items[i]}` : items[i]
const result = await apiGenerate({ model: 'stable-diffusion:sdxl', prompt })
lastResult = result.imageUrl
updateNode<IteratorNode>(this.editor, shape, (n) => ({
...n,
completedCount: i + 1,
lastResultUrl: lastResult,
}))
}
return { output: lastResult, current_item: items[items.length - 1] }
}
getOutputInfo(shape: NodeShape, node: IteratorNode, inputs: InfoValues): InfoValues {
const items = node.items
.split('\n')
.map((s) => s.trim())
.filter((s) => s.length > 0)
const currentItem =
node.completedCount > 0
? (items[Math.min(node.completedCount - 1, items.length - 1)] ?? null)
: null
return {
output: {
value: node.lastResultUrl,
isOutOfDate: areAnyInputsOutOfDate(inputs) || shape.props.isOutOfDate,
dataType: 'image',
},
current_item: {
value: currentItem,
isOutOfDate: areAnyInputsOutOfDate(inputs) || shape.props.isOutOfDate,
dataType: 'text',
},
}
}
Component = IteratorNodeComponent
}
function IteratorNodeComponent({ shape, node }: NodeComponentProps<IteratorNode>) {
const editor = useEditor()
const templateInput = useValue(
'template',
() => getNodeInputPortValues(editor, shape.id).template,
[editor, shape.id]
)
const items = node.items
.split('\n')
.map((s) => s.trim())
.filter((s) => s.length > 0)
return (
<>
<NodeRow>
<Port shapeId={shape.id} portId="template" />
<NodePortLabel dataType="any">Template</NodePortLabel>
{templateInput ? (
<span className="NodeRow-connected-value">connected</span>
) : (
<span className="NodeRow-disconnected">optional</span>
)}
</NodeRow>
<NodeRow className="PromptNode-row">
<textarea
className="PromptNode-textarea"
value={node.items}
placeholder="One item per line..."
onChange={(e) =>
updateNode<IteratorNode>(editor, shape, (n) => ({
...n,
items: e.target.value,
}))
}
onPointerDown={(e) => e.stopPropagation()}
/>
</NodeRow>
<NodeRow>
<span className="NodeInputRow-label" style={{ flex: 1 }}>
{node.totalCount > 0
? `${node.completedCount} / ${node.totalCount}`
: `${items.length} item${items.length !== 1 ? 's' : ''}`}
</span>
{node.totalCount > 0 && node.completedCount < node.totalCount && (
<div
style={{
flex: 2,
height: 4,
background: 'var(--tl-color-muted-0)',
borderRadius: 2,
overflow: 'hidden',
}}
>
<div
style={{
width: `${(node.completedCount / node.totalCount) * 100}%`,
height: '100%',
background: 'var(--tl-color-selected)',
transition: 'width 0.3s ease',
}}
/>
</div>
)}
</NodeRow>
<div
className={classNames('NodeImagePreview', {
NodeImagePreview_loading: shape.props.isOutOfDate,
})}
>
{node.lastResultUrl ? (
<NodeImage src={node.lastResultUrl} alt="Iterator result" />
) : (
<div className="NodeImagePreview-empty">
<span>Run to iterate items</span>
</div>
)}
</div>
</>
)
}

View File

@@ -0,0 +1,194 @@
import { useCallback, useState } from 'react'
import { T, useEditor } from 'tldraw'
import { LoadImageIcon } from '../../components/icons/LoadImageIcon'
import {
NODE_HEADER_HEIGHT_PX,
NODE_IMAGE_PREVIEW_HEIGHT_PX,
NODE_ROW_HEIGHT_PX,
NODE_WIDTH_PX,
} from '../../constants'
import { ShapePort } from '../../ports/Port'
import { sleep } from '../../utils/sleep'
import { NodeShape } from '../NodeShapeUtil'
import {
ExecutionResult,
InfoValues,
NodeComponentProps,
NodeDefinition,
NodeImage,
NodeRow,
updateNode,
} from './shared'
export type LoadImageNode = T.TypeOf<typeof LoadImageNode>
export const LoadImageNode = T.object({
type: T.literal('load_image'),
imageUrl: T.string.nullable(),
})
export class LoadImageNodeDefinition extends NodeDefinition<LoadImageNode> {
static type = 'load_image'
static validator = LoadImageNode
title = 'Load image'
heading = 'Image'
icon = <LoadImageIcon />
category = 'input'
getDefault(): LoadImageNode {
return {
type: 'load_image',
imageUrl: null,
}
}
getBodyHeightPx() {
return NODE_ROW_HEIGHT_PX + NODE_IMAGE_PREVIEW_HEIGHT_PX
}
getPorts(): Record<string, ShapePort> {
return {
output: {
id: 'output',
x: NODE_WIDTH_PX,
y: NODE_HEADER_HEIGHT_PX / 2,
terminal: 'start',
dataType: 'image',
},
}
}
async execute(_shape: NodeShape, node: LoadImageNode): Promise<ExecutionResult> {
await sleep(300)
return { output: node.imageUrl }
}
getOutputInfo(shape: NodeShape, node: LoadImageNode): InfoValues {
return {
output: {
value: node.imageUrl,
isOutOfDate: shape.props.isOutOfDate,
dataType: 'image',
},
}
}
Component = LoadImageNodeComponent
}
/** Read a File as a data URL. */
function readFileAsDataUrl(file: File): Promise<string> {
return new Promise((resolve, reject) => {
const reader = new FileReader()
reader.onload = () => resolve(reader.result as string)
reader.onerror = () => reject(reader.error)
reader.readAsDataURL(file)
})
}
/** Open the native file picker for an image file. */
function selectImageFile(): Promise<File | null> {
return new Promise((resolve) => {
const input = document.createElement('input')
input.type = 'file'
input.accept = 'image/*'
input.style.display = 'none'
const dispose = () => {
input.removeEventListener('change', onChange)
input.removeEventListener('cancel', onCancel)
input.remove()
}
const onChange = (event: Event) => {
const fileList = (event.target as HTMLInputElement).files
resolve(fileList && fileList.length > 0 ? fileList[0] : null)
dispose()
}
const onCancel = () => {
resolve(null)
dispose()
}
document.body.appendChild(input)
input.addEventListener('cancel', onCancel)
input.addEventListener('change', onChange)
input.click()
})
}
function LoadImageNodeComponent({ shape, node }: NodeComponentProps<LoadImageNode>) {
const editor = useEditor()
const [isDragOver, setIsDragOver] = useState(false)
const handleFile = useCallback(
async (file: File) => {
if (!file.type.startsWith('image/')) return
const dataUrl = await readFileAsDataUrl(file)
updateNode<LoadImageNode>(editor, shape, (n) => ({ ...n, imageUrl: dataUrl }))
},
[editor, shape]
)
const handleBrowse = useCallback(async () => {
const file = await selectImageFile()
if (file) handleFile(file)
}, [handleFile])
const handleDrop = useCallback(
(e: React.DragEvent) => {
e.preventDefault()
e.stopPropagation()
setIsDragOver(false)
const file = e.dataTransfer.files[0]
if (file) handleFile(file)
},
[handleFile]
)
const handleDragOver = useCallback((e: React.DragEvent) => {
e.preventDefault()
e.stopPropagation()
setIsDragOver(true)
}, [])
const handleDragLeave = useCallback((e: React.DragEvent) => {
e.preventDefault()
e.stopPropagation()
setIsDragOver(false)
}, [])
return (
<>
<NodeRow>
<button
className="LoadImageNode-browse"
onClick={handleBrowse}
onPointerDown={(e) => e.stopPropagation()}
>
{node.imageUrl ? 'Replace...' : 'Browse...'}
</button>
{node.imageUrl && (
<button
className="LoadImageNode-clear"
onClick={() =>
updateNode<LoadImageNode>(editor, shape, (n) => ({ ...n, imageUrl: null }))
}
onPointerDown={(e) => e.stopPropagation()}
title="Clear image"
>
×
</button>
)}
</NodeRow>
<div
className={`NodeImagePreview ${isDragOver ? 'NodeImagePreview_dragover' : ''}`}
onDrop={handleDrop}
onDragOver={handleDragOver}
onDragLeave={handleDragLeave}
>
{node.imageUrl ? (
<NodeImage src={node.imageUrl} alt="Loaded" />
) : (
<div className="NodeImagePreview-empty">
<span>{isDragOver ? 'Drop image here' : 'Drop or browse for an image'}</span>
</div>
)}
</div>
</>
)
}

View File

@@ -0,0 +1,140 @@
import { T, useEditor } from 'tldraw'
import { ModelIcon } from '../../components/icons/ModelIcon'
import { NODE_HEADER_HEIGHT_PX, NODE_ROW_HEIGHT_PX, NODE_WIDTH_PX } from '../../constants'
import { ShapePort } from '../../ports/Port'
import { sleep } from '../../utils/sleep'
import { NodeShape } from '../NodeShapeUtil'
import {
ExecutionResult,
InfoValues,
NodeComponentProps,
NodeDefinition,
NodeRow,
updateNode,
} from './shared'
interface ModelInfo {
id: string
label: string
}
const PROVIDERS: Record<string, { label: string; models: ModelInfo[] }> = {
flux: {
label: 'Flux',
models: [
{ id: 'flux-dev', label: 'Flux Dev' },
{ id: 'flux-schnell', label: 'Flux Schnell' },
{ id: 'flux-pro', label: 'Flux Pro' },
],
},
google: {
label: 'Google',
models: [
{ id: 'nano-banana-pro', label: 'Nano Banana Pro' },
{ id: 'nano-banana', label: 'Nano Banana' },
{ id: 'imagen-4-fast', label: 'Imagen 4 Fast' },
],
},
}
export type ModelNode = T.TypeOf<typeof ModelNode>
export const ModelNode = T.object({
type: T.literal('model'),
provider: T.string,
modelId: T.string,
})
export class ModelNodeDefinition extends NodeDefinition<ModelNode> {
static type = 'model'
static validator = ModelNode
title = 'Load model'
heading = 'Model'
icon = <ModelIcon />
category = 'input'
getDefault(): ModelNode {
return {
type: 'model',
provider: 'flux',
modelId: 'flux-dev',
}
}
getBodyHeightPx() {
return NODE_ROW_HEIGHT_PX * 2
}
getPorts(): Record<string, ShapePort> {
return {
output: {
id: 'output',
x: NODE_WIDTH_PX,
y: NODE_HEADER_HEIGHT_PX / 2,
terminal: 'start',
dataType: 'model',
},
}
}
async execute(_shape: NodeShape, node: ModelNode): Promise<ExecutionResult> {
await sleep(500)
return { output: `${node.provider}:${node.modelId}` }
}
getOutputInfo(shape: NodeShape, node: ModelNode): InfoValues {
return {
output: {
value: `${node.provider}:${node.modelId}`,
isOutOfDate: shape.props.isOutOfDate,
dataType: 'model',
},
}
}
Component = ModelNodeComponent
}
function ModelNodeComponent({ shape, node }: NodeComponentProps<ModelNode>) {
const editor = useEditor()
const provider = PROVIDERS[node.provider] ?? PROVIDERS['flux']
const models = provider.models
return (
<>
<NodeRow>
<span className="NodeInputRow-label">Provider</span>
<select
value={node.provider}
onChange={(e) => {
const newProvider = e.target.value
const newModels = PROVIDERS[newProvider]?.models
const firstModel = newModels?.[0]?.id ?? ''
updateNode<ModelNode>(editor, shape, (n) => ({
...n,
provider: newProvider,
modelId: firstModel,
}))
}}
>
{Object.entries(PROVIDERS).map(([id, p]) => (
<option key={id} value={id}>
{p.label}
</option>
))}
</select>
</NodeRow>
<NodeRow>
<span className="NodeInputRow-label">Model</span>
<select
value={node.modelId}
onChange={(e) =>
updateNode<ModelNode>(editor, shape, (n) => ({
...n,
modelId: e.target.value,
}))
}
>
{models.map((m) => (
<option key={m.id} value={m.id}>
{m.label}
</option>
))}
</select>
</NodeRow>
</>
)
}

View File

@@ -0,0 +1,83 @@
import { T, TldrawUiSlider, useEditor } from 'tldraw'
import { NumberIcon } from '../../components/icons/NumberIcon'
import { NODE_HEADER_HEIGHT_PX, NODE_ROW_HEIGHT_PX, NODE_WIDTH_PX } from '../../constants'
import { ShapePort } from '../../ports/Port'
import { sleep } from '../../utils/sleep'
import { NodeShape } from '../NodeShapeUtil'
import {
ExecutionResult,
InfoValues,
NodeComponentProps,
NodeDefinition,
NodeRow,
updateNode,
} from './shared'
export type NumberNode = T.TypeOf<typeof NumberNode>
export const NumberNode = T.object({
type: T.literal('number'),
value: T.number,
})
export class NumberNodeDefinition extends NodeDefinition<NumberNode> {
static type = 'number'
static validator = NumberNode
title = 'Number'
heading = 'Number'
hidden = true as const
icon = <NumberIcon />
category = 'input'
getDefault(): NumberNode {
return {
type: 'number',
value: 50,
}
}
getBodyHeightPx() {
return NODE_ROW_HEIGHT_PX
}
getPorts(): Record<string, ShapePort> {
return {
output: {
id: 'output',
x: NODE_WIDTH_PX,
y: NODE_HEADER_HEIGHT_PX / 2,
terminal: 'start',
dataType: 'number',
},
}
}
async execute(_shape: NodeShape, node: NumberNode): Promise<ExecutionResult> {
await sleep(200)
return { output: node.value }
}
getOutputInfo(shape: NodeShape, node: NumberNode): InfoValues {
return {
output: {
value: node.value,
isOutOfDate: shape.props.isOutOfDate,
dataType: 'number',
},
}
}
Component = NumberNodeComponent
}
function NumberNodeComponent({ shape, node }: NodeComponentProps<NumberNode>) {
const editor = useEditor()
return (
<NodeRow className="NumberNode" onPointerDown={editor.markEventAsHandled}>
<TldrawUiSlider
steps={100}
value={node.value}
label="Value"
title={node.value.toString()}
onValueChange={(value) => {
editor.setSelectedShapes([shape.id])
updateNode<NumberNode>(editor, shape, (node) => ({ ...node, value }), false)
}}
onHistoryMark={() => {}}
/>
</NodeRow>
)
}

View File

@@ -0,0 +1,120 @@
import classNames from 'classnames'
import { T, useEditor, useValue } from 'tldraw'
import { PreviewIcon } from '../../components/icons/PreviewIcon'
import {
NODE_HEADER_HEIGHT_PX,
NODE_IMAGE_PREVIEW_HEIGHT_PX,
NODE_ROW_HEADER_GAP_PX,
NODE_ROW_HEIGHT_PX,
} from '../../constants'
import { Port, ShapePort } from '../../ports/Port'
import { sleep } from '../../utils/sleep'
import { getNodeInputPortValues } from '../nodePorts'
import { NodeShape } from '../NodeShapeUtil'
import {
ExecutionResult,
InfoValues,
InputValues,
NodeComponentProps,
NodeDefinition,
NodeImage,
NodePortLabel,
NodeRow,
STOP_EXECUTION,
updateNode,
} from './shared'
export type PreviewNode = T.TypeOf<typeof PreviewNode>
export const PreviewNode = T.object({
type: T.literal('preview'),
lastImageUrl: T.string.nullable(),
})
export class PreviewNodeDefinition extends NodeDefinition<PreviewNode> {
static type = 'preview'
static validator = PreviewNode
title = 'Preview'
heading = 'Preview'
icon = <PreviewIcon />
category = 'output'
resultKeys = ['lastImageUrl'] as const
getDefault(): PreviewNode {
return {
type: 'preview',
lastImageUrl: null,
}
}
getBodyHeightPx() {
return NODE_ROW_HEIGHT_PX + NODE_IMAGE_PREVIEW_HEIGHT_PX
}
getPorts(): Record<string, ShapePort> {
const baseY = NODE_HEADER_HEIGHT_PX + NODE_ROW_HEADER_GAP_PX
return {
image: {
id: 'image',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 0.5,
terminal: 'end',
dataType: 'image',
},
// No output port — this is a terminal/sink node
}
}
async execute(
shape: NodeShape,
_node: PreviewNode,
inputs: InputValues
): Promise<ExecutionResult> {
await sleep(200)
const imageUrl = inputs.image as string | null
updateNode<PreviewNode>(this.editor, shape, (n) => ({
...n,
lastImageUrl: imageUrl ?? null,
}))
return {}
}
getOutputInfo(): InfoValues {
return {}
}
Component = PreviewNodeComponent
}
function PreviewNodeComponent({ shape, node }: NodeComponentProps<PreviewNode>) {
const editor = useEditor()
const imageInput = useValue('image input', () => getNodeInputPortValues(editor, shape.id).image, [
editor,
shape.id,
])
const displayUrl =
imageInput && !imageInput.isOutOfDate && imageInput.value !== STOP_EXECUTION
? (imageInput.value as string)
: node.lastImageUrl
return (
<>
<NodeRow>
<Port shapeId={shape.id} portId="image" />
<NodePortLabel dataType="image">Image</NodePortLabel>
{imageInput ? (
<span className="NodeRow-connected-value">connected</span>
) : (
<span className="NodeRow-disconnected">not connected</span>
)}
</NodeRow>
<div
className={classNames('NodeImagePreview', {
NodeImagePreview_loading: shape.props.isOutOfDate,
})}
>
{displayUrl ? (
<NodeImage src={displayUrl} alt="Preview" />
) : (
<div className="NodeImagePreview-empty">
<span>No image to preview</span>
</div>
)}
</div>
</>
)
}

View File

@@ -0,0 +1,202 @@
import { T, useEditor, useValue } from 'tldraw'
import { PromptConcatIcon } from '../../components/icons/PromptConcatIcon'
import {
NODE_HEADER_HEIGHT_PX,
NODE_ROW_HEADER_GAP_PX,
NODE_ROW_HEIGHT_PX,
NODE_WIDTH_PX,
} from '../../constants'
import { Port, ShapePort } from '../../ports/Port'
import { sleep } from '../../utils/sleep'
import { getNodeInputPortValues } from '../nodePorts'
import { NodeShape } from '../NodeShapeUtil'
import {
areAnyInputsOutOfDate,
ExecutionResult,
InfoValues,
InputValues,
NodeComponentProps,
NodeDefinition,
NodePortLabel,
NodeRow,
updateNode,
} from './shared'
const SEPARATORS = [
{ id: 'space', label: 'Space', value: ' ' },
{ id: 'newline', label: 'Newline', value: '\n' },
{ id: 'comma', label: 'Comma', value: ', ' },
{ id: 'none', label: 'None', value: '' },
] as const
export type PromptConcatNode = T.TypeOf<typeof PromptConcatNode>
export const PromptConcatNode = T.object({
type: T.literal('prompt_concat'),
separator: T.string,
})
export class PromptConcatNodeDefinition extends NodeDefinition<PromptConcatNode> {
static type = 'prompt_concat'
static validator = PromptConcatNode
title = 'Concat'
heading = 'Prompt concat'
icon = <PromptConcatIcon />
category = 'process'
getDefault(): PromptConcatNode {
return {
type: 'prompt_concat',
separator: 'space',
}
}
getBodyHeightPx() {
// 3 input rows + separator row + preview row
return NODE_ROW_HEIGHT_PX * 5
}
getPorts(): Record<string, ShapePort> {
const baseY = NODE_HEADER_HEIGHT_PX + NODE_ROW_HEADER_GAP_PX
return {
prefix: {
id: 'prefix',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 0.5,
terminal: 'end',
dataType: 'text',
},
main: {
id: 'main',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 1.5,
terminal: 'end',
dataType: 'text',
},
suffix: {
id: 'suffix',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 2.5,
terminal: 'end',
dataType: 'text',
},
output: {
id: 'output',
x: NODE_WIDTH_PX,
y: NODE_HEADER_HEIGHT_PX / 2,
terminal: 'start',
dataType: 'text',
},
}
}
async execute(
_shape: NodeShape,
node: PromptConcatNode,
inputs: InputValues
): Promise<ExecutionResult> {
await sleep(100)
const sep = SEPARATORS.find((s) => s.id === node.separator)?.value ?? ' '
const parts = [inputs.prefix, inputs.main, inputs.suffix].filter(
(v): v is string => typeof v === 'string' && v.length > 0
)
return { output: parts.join(sep) }
}
getOutputInfo(shape: NodeShape, node: PromptConcatNode, inputs: InfoValues): InfoValues {
const sep = SEPARATORS.find((s) => s.id === node.separator)?.value ?? ' '
const parts: string[] = []
for (const key of ['prefix', 'main', 'suffix'] as const) {
const info = inputs[key]
if (info && typeof info.value === 'string' && info.value.length > 0) {
parts.push(info.value)
}
}
return {
output: {
value: parts.length > 0 ? parts.join(sep) : null,
isOutOfDate: areAnyInputsOutOfDate(inputs) || shape.props.isOutOfDate,
dataType: 'text',
},
}
}
Component = PromptConcatNodeComponent
}
function PromptConcatNodeComponent({ shape, node }: NodeComponentProps<PromptConcatNode>) {
const editor = useEditor()
const prefixInput = useValue('prefix', () => getNodeInputPortValues(editor, shape.id).prefix, [
editor,
shape.id,
])
const mainInput = useValue('main', () => getNodeInputPortValues(editor, shape.id).main, [
editor,
shape.id,
])
const suffixInput = useValue('suffix', () => getNodeInputPortValues(editor, shape.id).suffix, [
editor,
shape.id,
])
const sep = SEPARATORS.find((s) => s.id === node.separator)?.value ?? ' '
const parts = [prefixInput?.value, mainInput?.value, suffixInput?.value].filter(
(v): v is string => typeof v === 'string' && v.length > 0
)
const preview = parts.join(sep)
return (
<>
<NodeRow>
<Port shapeId={shape.id} portId="prefix" />
<NodePortLabel dataType="text">Prefix</NodePortLabel>
{prefixInput ? (
<span className="NodeRow-connected-value">
{typeof prefixInput.value === 'string' ? prefixInput.value.slice(0, 20) : 'connected'}
</span>
) : (
<span className="NodeRow-disconnected">not connected</span>
)}
</NodeRow>
<NodeRow>
<Port shapeId={shape.id} portId="main" />
<NodePortLabel dataType="text">Main</NodePortLabel>
{mainInput ? (
<span className="NodeRow-connected-value">
{typeof mainInput.value === 'string' ? mainInput.value.slice(0, 20) : 'connected'}
</span>
) : (
<span className="NodeRow-disconnected">not connected</span>
)}
</NodeRow>
<NodeRow>
<Port shapeId={shape.id} portId="suffix" />
<NodePortLabel dataType="text">Suffix</NodePortLabel>
{suffixInput ? (
<span className="NodeRow-connected-value">
{typeof suffixInput.value === 'string' ? suffixInput.value.slice(0, 20) : 'connected'}
</span>
) : (
<span className="NodeRow-disconnected">not connected</span>
)}
</NodeRow>
<NodeRow>
<span className="NodeInputRow-label">Sep</span>
<select
value={node.separator}
onChange={(e) =>
updateNode<PromptConcatNode>(editor, shape, (n) => ({
...n,
separator: e.target.value,
}))
}
>
{SEPARATORS.map((s) => (
<option key={s.id} value={s.id}>
{s.label}
</option>
))}
</select>
</NodeRow>
<NodeRow>
<span className="NodeRow-connected-value" style={{ fontSize: 10, opacity: 0.7 }}>
{preview ? (preview.length > 40 ? preview.slice(0, 38) + '...' : preview) : 'no inputs'}
</span>
</NodeRow>
</>
)
}

View File

@@ -0,0 +1,89 @@
import { T, useEditor } from 'tldraw'
import { PromptIcon } from '../../components/icons/PromptIcon'
import { NODE_HEADER_HEIGHT_PX, NODE_ROW_HEIGHT_PX, NODE_WIDTH_PX } from '../../constants'
import { ShapePort } from '../../ports/Port'
import { sleep } from '../../utils/sleep'
import { NodeShape } from '../NodeShapeUtil'
import {
ExecutionResult,
InfoValues,
NodeComponentProps,
NodeDefinition,
NodeRow,
updateNode,
} from './shared'
export type PromptNode = T.TypeOf<typeof PromptNode>
export const PromptNode = T.object({
type: T.literal('prompt'),
text: T.string,
})
export class PromptNodeDefinition extends NodeDefinition<PromptNode> {
static type = 'prompt'
static validator = PromptNode
title = 'Prompt'
heading = 'Prompt'
icon = <PromptIcon />
category = 'input'
getDefault(): PromptNode {
return {
type: 'prompt',
text: 'a photo of a cat sitting on a windowsill',
}
}
getBodyHeightPx() {
return NODE_ROW_HEIGHT_PX * 2
}
getPorts(): Record<string, ShapePort> {
return {
output: {
id: 'output',
x: NODE_WIDTH_PX,
y: NODE_HEADER_HEIGHT_PX / 2,
terminal: 'start',
dataType: 'text',
},
}
}
async execute(_shape: NodeShape, node: PromptNode): Promise<ExecutionResult> {
await sleep(200)
return { output: node.text }
}
getOutputInfo(shape: NodeShape, node: PromptNode): InfoValues {
return {
output: {
value: node.text,
isOutOfDate: shape.props.isOutOfDate,
dataType: 'text',
},
}
}
Component = PromptNodeComponent
}
function PromptNodeComponent({ shape, node }: NodeComponentProps<PromptNode>) {
const editor = useEditor()
return (
<NodeRow className="PromptNode-row">
<textarea
className="PromptNode-textarea"
value={node.text}
placeholder="Enter your prompt..."
onChange={(e) =>
updateNode<PromptNode>(
editor,
shape,
(n) => ({
...n,
text: e.target.value,
}),
false
)
}
onPointerDown={(e) => e.stopPropagation()}
onFocus={() => editor.setSelectedShapes([shape.id])}
/>
</NodeRow>
)
}

View File

@@ -0,0 +1,122 @@
import { T } from 'tldraw'
import { RouterIcon } from '../../components/icons/RouterIcon'
import {
NODE_HEADER_HEIGHT_PX,
NODE_ROW_HEADER_GAP_PX,
NODE_ROW_HEIGHT_PX,
NODE_WIDTH_PX,
} from '../../constants'
import { ShapePort } from '../../ports/Port'
import { NodeShape } from '../NodeShapeUtil'
import {
ExecutionResult,
InfoValues,
InputValues,
NodeComponentProps,
NodeDefinition,
NodePortLabel,
NodeRow,
} from './shared'
export type RouterNode = T.TypeOf<typeof RouterNode>
export const RouterNode = T.object({
type: T.literal('router'),
outputCount: T.number,
})
export class RouterNodeDefinition extends NodeDefinition<RouterNode> {
static type = 'router'
static validator = RouterNode
title = 'Router'
heading = 'Router'
hidden = true as const
icon = <RouterIcon />
category = 'utility'
getDefault(): RouterNode {
return {
type: 'router',
outputCount: 3,
}
}
getBodyHeightPx(_shape: NodeShape, node: RouterNode) {
// 1 input row + N output rows
return NODE_ROW_HEIGHT_PX * (1 + node.outputCount)
}
getPorts(_shape: NodeShape, node: RouterNode): Record<string, ShapePort> {
const baseY = NODE_HEADER_HEIGHT_PX + NODE_ROW_HEADER_GAP_PX
const ports: Record<string, ShapePort> = {
input: {
id: 'input',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 0.5,
terminal: 'end',
dataType: 'any',
},
}
for (let i = 0; i < node.outputCount; i++) {
ports[`out_${i}`] = {
id: `out_${i}`,
x: NODE_WIDTH_PX,
y: baseY + NODE_ROW_HEIGHT_PX * (1.5 + i),
terminal: 'start',
dataType: 'any',
}
}
return ports
}
async execute(
_shape: NodeShape,
node: RouterNode,
inputs: InputValues
): Promise<ExecutionResult> {
const value = (inputs.input as string | number | null) ?? null
const result: ExecutionResult = {}
for (let i = 0; i < node.outputCount; i++) {
result[`out_${i}`] = value
}
return result
}
getOutputInfo(shape: NodeShape, node: RouterNode, inputs: InfoValues): InfoValues {
const inputInfo = inputs.input
const result: InfoValues = {}
for (let i = 0; i < node.outputCount; i++) {
if (inputInfo) {
// Router input is always single; forward the value to each output.
const singleValue = inputInfo.multi ? inputInfo.value[0] : inputInfo.value
result[`out_${i}`] = {
value: singleValue,
isOutOfDate: inputInfo.isOutOfDate || shape.props.isOutOfDate,
dataType: inputInfo.dataType,
}
} else {
result[`out_${i}`] = {
value: null,
isOutOfDate: shape.props.isOutOfDate,
dataType: 'any',
}
}
}
return result
}
Component = RouterNodeComponent
}
function RouterNodeComponent({ node }: NodeComponentProps<RouterNode>) {
return (
<>
<NodeRow>
<span
className="Port Port_end"
style={{ '--port-color': 'var(--port-color-any)' } as React.CSSProperties}
/>
<NodePortLabel dataType="any">Input</NodePortLabel>
<span className="NodeRow-connected-value">{node.outputCount} outputs</span>
</NodeRow>
{Array.from({ length: node.outputCount }, (_, i) => (
<NodeRow key={i}>
<NodePortLabel dataType="any">Out {i + 1}</NodePortLabel>
</NodeRow>
))}
</>
)
}

View File

@@ -0,0 +1,253 @@
import classNames from 'classnames'
import { T, useEditor, useValue } from 'tldraw'
import { apiStyleTransfer } from '../../api/pipelineApi'
import { StyleTransferIcon } from '../../components/icons/StyleTransferIcon'
import {
NODE_HEADER_HEIGHT_PX,
NODE_IMAGE_PREVIEW_HEIGHT_PX,
NODE_ROW_HEADER_GAP_PX,
NODE_ROW_HEIGHT_PX,
NODE_WIDTH_PX,
} from '../../constants'
import { Port, ShapePort } from '../../ports/Port'
import { getNodeInputPortValues } from '../nodePorts'
import { NodeShape } from '../NodeShapeUtil'
import {
areAnyInputsOutOfDate,
ExecutionResult,
InfoValues,
InputValues,
NodeComponentProps,
NodeDefinition,
NodeImage,
NodePlaceholder,
NodePortLabel,
NodeRow,
STOP_EXECUTION,
updateNode,
} from './shared'
const STYLE_MODELS = [
{ id: 'fast', label: 'Fast' },
{ id: 'high-quality', label: 'High quality' },
{ id: 'realistic', label: 'Realistic' },
{ id: 'cinematic', label: 'Cinematic' },
{ id: 'animated', label: 'Animated' },
] as const
export type StyleTransferNode = T.TypeOf<typeof StyleTransferNode>
export const StyleTransferNode = T.object({
type: T.literal('style_transfer'),
model: T.string,
strength: T.number,
lastResultUrl: T.string.nullable(),
})
export class StyleTransferNodeDefinition extends NodeDefinition<StyleTransferNode> {
static type = 'style_transfer'
static validator = StyleTransferNode
title = 'Style transfer'
heading = 'Style transfer'
icon = <StyleTransferIcon />
category = 'process'
resultKeys = ['lastResultUrl'] as const
getDefault(): StyleTransferNode {
return {
type: 'style_transfer',
model: 'fast',
strength: 50,
lastResultUrl: null,
}
}
getBodyHeightPx() {
// style input + content input + prompt input + model row + strength row + preview
return NODE_ROW_HEIGHT_PX * 5 + NODE_IMAGE_PREVIEW_HEIGHT_PX
}
getPorts(): Record<string, ShapePort> {
const baseY = NODE_HEADER_HEIGHT_PX + NODE_ROW_HEADER_GAP_PX
return {
style: {
id: 'style',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 0.5,
terminal: 'end',
dataType: 'image',
},
content: {
id: 'content',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 1.5,
terminal: 'end',
dataType: 'image',
},
prompt: {
id: 'prompt',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 2.5,
terminal: 'end',
dataType: 'text',
},
output: {
id: 'output',
x: NODE_WIDTH_PX,
y: NODE_HEADER_HEIGHT_PX / 2,
terminal: 'start',
dataType: 'image',
},
}
}
async execute(
shape: NodeShape,
node: StyleTransferNode,
inputs: InputValues
): Promise<ExecutionResult> {
const styleUrl = (inputs.style as string) ?? ''
const contentUrl = (inputs.content as string) ?? undefined
const prompt = (inputs.prompt as string) ?? undefined
const result = await apiStyleTransfer({
styleImageUrl: styleUrl,
contentImageUrl: contentUrl,
prompt,
model: node.model,
strength: node.strength / 100,
})
updateNode<StyleTransferNode>(this.editor, shape, (n) => ({
...n,
lastResultUrl: result.imageUrl,
}))
return { output: result.imageUrl }
}
getOutputInfo(shape: NodeShape, node: StyleTransferNode, inputs: InfoValues): InfoValues {
return {
output: {
value: node.lastResultUrl,
isOutOfDate: areAnyInputsOutOfDate(inputs) || shape.props.isOutOfDate,
dataType: 'image',
},
}
}
Component = StyleTransferNodeComponent
}
function StyleTransferNodeComponent({ shape, node }: NodeComponentProps<StyleTransferNode>) {
const editor = useEditor()
const styleInput = useValue('style input', () => getNodeInputPortValues(editor, shape.id).style, [
editor,
shape.id,
])
const contentInput = useValue(
'content input',
() => getNodeInputPortValues(editor, shape.id).content,
[editor, shape.id]
)
const promptInput = useValue(
'prompt input',
() => getNodeInputPortValues(editor, shape.id).prompt,
[editor, shape.id]
)
return (
<>
<NodeRow>
<Port shapeId={shape.id} portId="style" />
<NodePortLabel dataType="image">Style</NodePortLabel>
{styleInput ? (
<span className="NodeRow-connected-value">
{styleInput.isOutOfDate || styleInput.value === STOP_EXECUTION ? (
<NodePlaceholder />
) : (
'connected'
)}
</span>
) : (
<span className="NodeRow-disconnected">not connected</span>
)}
</NodeRow>
<NodeRow>
<Port shapeId={shape.id} portId="content" />
<NodePortLabel dataType="image">Content</NodePortLabel>
{contentInput ? (
<span className="NodeRow-connected-value">
{contentInput.isOutOfDate || contentInput.value === STOP_EXECUTION ? (
<NodePlaceholder />
) : (
'connected'
)}
</span>
) : (
<span className="NodeRow-disconnected">optional</span>
)}
</NodeRow>
<NodeRow>
<Port shapeId={shape.id} portId="prompt" />
<NodePortLabel dataType="text">Prompt</NodePortLabel>
{promptInput ? (
<span className="NodeRow-connected-value">
{promptInput.isOutOfDate || promptInput.value === STOP_EXECUTION ? (
<NodePlaceholder />
) : typeof promptInput.value === 'string' ? (
promptInput.value.slice(0, 15) + (promptInput.value.length > 15 ? '...' : '')
) : (
'connected'
)}
</span>
) : (
<span className="NodeRow-disconnected">optional</span>
)}
</NodeRow>
<NodeRow>
<span className="NodeInputRow-label">Model</span>
<select
value={node.model}
onChange={(e) =>
updateNode<StyleTransferNode>(editor, shape, (n) => ({
...n,
model: e.target.value,
}))
}
>
{STYLE_MODELS.map((m) => (
<option key={m.id} value={m.id}>
{m.label}
</option>
))}
</select>
</NodeRow>
<NodeRow className="NodeInputRow">
<span className="NodeInputRow-label">Strength</span>
<input
type="range"
min="0"
max="100"
value={node.strength}
onChange={(e) =>
updateNode<StyleTransferNode>(
editor,
shape,
(n) => ({ ...n, strength: Number(e.target.value) }),
false
)
}
onPointerDown={(e) => e.stopPropagation()}
/>
<span className="NodeRow-value">{node.strength}%</span>
</NodeRow>
<div
className={classNames('NodeImagePreview', {
NodeImagePreview_loading: shape.props.isOutOfDate,
})}
>
{node.lastResultUrl ? (
<NodeImage src={node.lastResultUrl} alt="Style transfer result" />
) : (
<div className="NodeImagePreview-empty">
<span>Connect a style image</span>
</div>
)}
</div>
</>
)
}

View File

@@ -0,0 +1,194 @@
import classNames from 'classnames'
import { T, useEditor, useValue } from 'tldraw'
import { apiUpscale } from '../../api/pipelineApi'
import { UpscaleIcon } from '../../components/icons/UpscaleIcon'
import {
NODE_HEADER_HEIGHT_PX,
NODE_IMAGE_PREVIEW_HEIGHT_PX,
NODE_ROW_HEADER_GAP_PX,
NODE_ROW_HEIGHT_PX,
NODE_WIDTH_PX,
} from '../../constants'
import { Port, ShapePort } from '../../ports/Port'
import { getNodeInputPortValues } from '../nodePorts'
import { NodeShape } from '../NodeShapeUtil'
import {
areAnyInputsOutOfDate,
ExecutionResult,
InfoValues,
InputValues,
NodeComponentProps,
NodeDefinition,
NodeImage,
NodePlaceholder,
NodePortLabel,
NodeRow,
STOP_EXECUTION,
updateNode,
} from './shared'
const SCALE_FACTORS = [
{ id: '2', label: '2x' },
{ id: '4', label: '4x' },
] as const
const METHODS = [
{ id: 'bilinear', label: 'Bilinear' },
{ id: 'lanczos', label: 'Lanczos' },
{ id: 'ai_enhanced', label: 'AI enhanced' },
] as const
export type UpscaleNode = T.TypeOf<typeof UpscaleNode>
export const UpscaleNode = T.object({
type: T.literal('upscale'),
scale: T.string,
method: T.string,
lastResultUrl: T.string.nullable(),
})
export class UpscaleNodeDefinition extends NodeDefinition<UpscaleNode> {
static type = 'upscale'
static validator = UpscaleNode
title = 'Upscale'
heading = 'Upscale'
hidden = true as const
icon = <UpscaleIcon />
category = 'process'
resultKeys = ['lastResultUrl'] as const
getDefault(): UpscaleNode {
return {
type: 'upscale',
scale: '2',
method: 'lanczos',
lastResultUrl: null,
}
}
getBodyHeightPx() {
return NODE_ROW_HEIGHT_PX * 3 + NODE_IMAGE_PREVIEW_HEIGHT_PX
}
getPorts(): Record<string, ShapePort> {
const baseY = NODE_HEADER_HEIGHT_PX + NODE_ROW_HEADER_GAP_PX
return {
image: {
id: 'image',
x: 0,
y: baseY + NODE_ROW_HEIGHT_PX * 0.5,
terminal: 'end',
dataType: 'image',
},
output: {
id: 'output',
x: NODE_WIDTH_PX,
y: NODE_HEADER_HEIGHT_PX / 2,
terminal: 'start',
dataType: 'image',
},
}
}
async execute(
shape: NodeShape,
node: UpscaleNode,
inputs: InputValues
): Promise<ExecutionResult> {
const imageUrl = (inputs.image as string) ?? ''
const result = await apiUpscale({
imageUrl,
scale: Number(node.scale) || 2,
method: node.method,
})
updateNode<UpscaleNode>(this.editor, shape, (n) => ({
...n,
lastResultUrl: result.imageUrl,
}))
return { output: result.imageUrl }
}
getOutputInfo(shape: NodeShape, node: UpscaleNode, inputs: InfoValues): InfoValues {
return {
output: {
value: node.lastResultUrl,
isOutOfDate: areAnyInputsOutOfDate(inputs) || shape.props.isOutOfDate,
dataType: 'image',
},
}
}
Component = UpscaleNodeComponent
}
function UpscaleNodeComponent({ shape, node }: NodeComponentProps<UpscaleNode>) {
const editor = useEditor()
const imageInput = useValue('image input', () => getNodeInputPortValues(editor, shape.id).image, [
editor,
shape.id,
])
return (
<>
<NodeRow>
<Port shapeId={shape.id} portId="image" />
<NodePortLabel dataType="image">Image</NodePortLabel>
{imageInput ? (
<span className="NodeRow-connected-value">
{imageInput.isOutOfDate || imageInput.value === STOP_EXECUTION ? (
<NodePlaceholder />
) : (
'connected'
)}
</span>
) : (
<span className="NodeRow-disconnected">not connected</span>
)}
</NodeRow>
<NodeRow>
<span className="NodeInputRow-label">Scale</span>
<select
value={node.scale}
onChange={(e) =>
updateNode<UpscaleNode>(editor, shape, (n) => ({
...n,
scale: e.target.value,
}))
}
>
{SCALE_FACTORS.map((s) => (
<option key={s.id} value={s.id}>
{s.label}
</option>
))}
</select>
</NodeRow>
<NodeRow>
<span className="NodeInputRow-label">Method</span>
<select
value={node.method}
onChange={(e) =>
updateNode<UpscaleNode>(editor, shape, (n) => ({
...n,
method: e.target.value,
}))
}
>
{METHODS.map((m) => (
<option key={m.id} value={m.id}>
{m.label}
</option>
))}
</select>
</NodeRow>
<div
className={classNames('NodeImagePreview', {
NodeImagePreview_loading: shape.props.isOutOfDate,
})}
>
{node.lastResultUrl ? (
<NodeImage src={node.lastResultUrl} alt="Upscaled" />
) : (
<div className="NodeImagePreview-empty">
<span>Connect an image to upscale</span>
</div>
)}
</div>
</>
)
}

View File

@@ -0,0 +1,393 @@
import classNames from 'classnames'
import { PointerEvent, SyntheticEvent, useCallback, useRef, useState } from 'react'
import {
Editor,
T,
TldrawUiButton,
TldrawUiButtonIcon,
TLShapeId,
TLUiIconJsx,
useEditor,
useValue,
} from 'tldraw'
import { AddIcon } from '../../components/icons/AddIcon'
import { SubtractIcon } from '../../components/icons/SubtractIcon'
import { NODE_WIDTH_PX, PORT_TYPE_COLORS, PortDataType } from '../../constants'
import { Port, PortId, ShapePort } from '../../ports/Port'
import { getNodeInputPortValues } from '../nodePorts'
import { NodeShape } from '../NodeShapeUtil'
import { NodeType } from '../nodeTypes'
/**
* Pipeline values can be strings (prompts, image URLs, model IDs), numbers (steps, cfg scale),
* or null (no value yet).
*/
export type PipelineValue = string | number | null
/**
* A special value that can be returned from a node to indicate that execution should stop.
*/
export type STOP_EXECUTION = typeof STOP_EXECUTION
export const STOP_EXECUTION = Symbol('STOP_EXECUTION')
export interface SingleInfoValue {
value: PipelineValue | STOP_EXECUTION
isOutOfDate: boolean
dataType: PortDataType
multi?: false
}
export interface MultiInfoValue {
value: (PipelineValue | STOP_EXECUTION)[]
isOutOfDate: boolean
dataType: PortDataType
multi: true
}
export type InfoValue = SingleInfoValue | MultiInfoValue
export function isMultiInfoValue(v: InfoValue): v is MultiInfoValue {
return v.multi === true
}
export interface InfoValues {
[key: string]: InfoValue
}
export interface ExecutionResult {
[key: string]: PipelineValue | STOP_EXECUTION
}
export interface InputValues {
[key: string]: PipelineValue | PipelineValue[]
}
export interface NodeComponentProps<Node extends { type: string }> {
shape: NodeShape
node: Node
}
export abstract class NodeDefinition<Node extends { type: string }> {
constructor(public readonly editor: Editor) {
const ctor = this.constructor as NodeDefinitionConstructor<Node>
this.type = ctor.type
this.validator = ctor.validator
}
readonly type: Node['type']
readonly validator: T.Validator<Node>
abstract readonly title: string
abstract readonly heading?: string
abstract readonly icon: TLUiIconJsx
/** A short category label for grouping in the toolbar. */
abstract readonly category: string
readonly resultKeys?: readonly string[]
readonly canResizeNode: boolean = false
/** If true, this node type is hidden from the toolbar and on-canvas picker. */
readonly hidden: boolean = false
getWidthPx(_shape: NodeShape, _node: Node): number {
return NODE_WIDTH_PX
}
abstract getDefault(): Node
abstract getBodyHeightPx(shape: NodeShape, node: Node): number
abstract getPorts(shape: NodeShape, node: Node): Record<string, ShapePort>
onPortConnect(_shape: NodeShape, _node: Node, _port: PortId): void {}
onPortDisconnect(_shape: NodeShape, _node: Node, _port: PortId): void {}
abstract getOutputInfo(shape: NodeShape, node: Node, inputs: InfoValues): InfoValues
abstract execute(shape: NodeShape, node: Node, inputs: InputValues): Promise<ExecutionResult>
abstract Component: React.ComponentType<NodeComponentProps<Node>>
}
export interface NodeDefinitionConstructor<Node extends { type: string }> {
new (editor: Editor): NodeDefinition<Node>
readonly type: Node['type']
readonly validator: T.Validator<Node>
}
/**
* Update the `node` prop within a node shape.
*/
export function updateNode<T extends NodeType>(
editor: Editor,
shape: NodeShape,
update: (node: T) => T,
isOutOfDate: boolean = true
) {
editor.updateShape({
id: shape.id,
type: shape.type,
props: { node: update(shape.props.node as T), isOutOfDate },
})
}
/**
* A row in a node. This component just applies some styling.
*/
export function NodeRow({
children,
className,
...props
}: {
children: React.ReactNode
className?: string
} & React.HTMLAttributes<HTMLDivElement>) {
return (
<div {...props} className={classNames('NodeRow', className)}>
{children}
</div>
)
}
/**
* A label for a port row, displayed next to the port.
*/
export function NodePortLabel({
children,
dataType,
}: {
children: React.ReactNode
dataType: PortDataType
}) {
return (
<span className="NodePortLabel" style={{ color: PORT_TYPE_COLORS[dataType] }}>
{children}
</span>
)
}
/**
* A row in a node for a numeric input. If the port is connected, the input is disabled and the
* value is taken from the port. Otherwise, the input is editable with a spinner for incrementing
* and decrementing the value.
*/
export function NodeInputRow({
shapeId,
portId,
label,
value,
onChange,
onBlur,
}: {
shapeId: TLShapeId
portId: PortId
label?: string
value: number
onChange: (value: number) => void
onBlur?: () => void
}) {
const editor = useEditor()
const inputRef = useRef<HTMLInputElement>(null)
const portInfo = useValue('from port', () => getNodeInputPortValues(editor, shapeId)[portId], [
editor,
shapeId,
portId,
])
const valueFromPort = portInfo?.value
const isOutOfDate = portInfo?.isOutOfDate
const [pendingValue, setPendingValue] = useState<string | null>(null)
const onPointerDown = useCallback((event: PointerEvent) => {
event.stopPropagation()
}, [])
const onSpinner = (delta: number) => {
const newValue = value + delta
onChange(newValue)
setPendingValue(String(newValue))
inputRef.current?.focus()
}
const displayValue = isOutOfDate
? '...'
: typeof valueFromPort === 'number'
? valueFromPort
: valueFromPort != null
? String(valueFromPort)
: (pendingValue ?? value)
return (
<NodeRow className="NodeInputRow">
<Port shapeId={shapeId} portId={portId} />
{label && <span className="NodeInputRow-label">{label}</span>}
{isOutOfDate || valueFromPort === STOP_EXECUTION ? (
<NodePlaceholder />
) : (
<input
ref={inputRef}
type="text"
inputMode="decimal"
disabled={valueFromPort != null}
value={displayValue}
onChange={(e) => {
setPendingValue(e.currentTarget.value)
const asNumber = Number(e.currentTarget.value.trim())
if (Number.isNaN(asNumber)) return
onChange(asNumber)
}}
onPointerDown={onPointerDown}
onBlur={() => {
setPendingValue(null)
onBlur?.()
}}
onFocus={() => {
editor.setSelectedShapes([shapeId])
}}
/>
)}
<div className="NodeInputRow-buttons">
<TldrawUiButton
title="decrement"
type="icon"
onPointerDown={onPointerDown}
onClick={() => onSpinner(-1)}
>
<TldrawUiButtonIcon icon={<SubtractIcon />} />
</TldrawUiButton>
<TldrawUiButton
title="increment"
type="icon"
onPointerDown={onPointerDown}
onClick={() => onSpinner(1)}
>
<TldrawUiButtonIcon icon={<AddIcon />} />
</TldrawUiButton>
</div>
</NodeRow>
)
}
/**
* A placeholder for a value that is not yet computed.
*/
export function NodePlaceholder() {
return <div className="NodeValue_placeholder" />
}
/**
* An image element that hides itself if the source fails to load.
*/
export function NodeImage({ src, alt }: { src: string; alt: string }) {
const onError = useCallback((e: SyntheticEvent<HTMLImageElement>) => {
e.currentTarget.style.display = 'none'
}, [])
return <img src={src} alt={alt} onError={onError} />
}
/**
* Format a pipeline value for display.
*/
export function NodeValue({ value }: { value: PipelineValue | STOP_EXECUTION }) {
if (value === STOP_EXECUTION || value === null) {
return <NodePlaceholder />
}
if (typeof value === 'number') {
return <>{formatNumber(value)}</>
}
// For strings, truncate long values
const str = String(value)
if (str.length > 20) {
return <span title={str}>{str.slice(0, 18)}...</span>
}
return <>{str}</>
}
function formatNumber(value: number): string {
if (value === 0) return '0'
if (!isFinite(value)) return value.toString()
const absValue = Math.abs(value)
const sign = value < 0 ? '-' : ''
if (absValue >= 1_000_000) {
return sign + (absValue / 1_000_000).toPrecision(3) + 'M'
}
if (absValue >= 1_000) {
return sign + (absValue / 1_000).toPrecision(3) + 'k'
}
if (absValue >= 1) {
return sign + absValue.toPrecision(5).replace(/\.?0+$/, '')
} else if (absValue >= 0.001) {
return sign + absValue.toPrecision(3)
} else {
return value.toExponential(2)
}
}
export function areAnyInputsOutOfDate(inputs: InfoValues): boolean {
return Object.values(inputs).some((input) => input.isOutOfDate)
}
/**
* Load a URL (data URL or http URL) into an HTMLImageElement.
*/
export function loadImage(url: string): Promise<HTMLImageElement> {
return new Promise((resolve, reject) => {
const img = new Image()
img.onload = () => resolve(img)
img.onerror = (_e) => reject(new Error('Failed to load image'))
img.crossOrigin = 'anonymous'
img.src = url
})
}
/**
* Convert a Blob to a data URL via FileReader.
*/
export function blobToDataUrl(blob: Blob): Promise<string> {
return new Promise((resolve, reject) => {
const reader = new FileReader()
reader.onloadend = () => resolve(reader.result as string)
reader.onerror = reject
reader.readAsDataURL(blob)
})
}
// ---------------------------------------------------------------------------
// Input coercion helpers
// ---------------------------------------------------------------------------
/** Coerce any pipeline value to a string. */
export function coerceToText(value: PipelineValue, fallback = ''): string {
if (value == null) return fallback
if (typeof value === 'number') return String(value)
return value
}
/** Coerce any pipeline value to a number. */
export function coerceToNumber(value: PipelineValue, fallback = 0): number {
if (value == null) return fallback
if (typeof value === 'number') return value
const n = parseFloat(value)
return Number.isNaN(n) ? fallback : n
}
/** Extract a single value from an InputValues entry (takes first element if array). */
export function getInput(inputs: InputValues, key: string): PipelineValue {
const v = inputs[key]
if (Array.isArray(v)) return v[0] ?? null
return v ?? null
}
/** Always return an array from an InputValues entry. */
export function getInputMulti(inputs: InputValues, key: string): PipelineValue[] {
const v = inputs[key]
if (v == null) return []
if (Array.isArray(v)) return v
return [v]
}
/** Extract a single value and coerce to string. */
export function getInputText(inputs: InputValues, key: string, fallback = ''): string {
return coerceToText(getInput(inputs, key), fallback)
}
/** Extract a single value and coerce to number. */
export function getInputNumber(inputs: InputValues, key: string, fallback = 0): number {
return coerceToNumber(getInput(inputs, key), fallback)
}

View File

@@ -0,0 +1,180 @@
import { createShapeId, StateNode, TLPointerEventInfo, TLShapeId } from 'tldraw'
import { onCanvasNodePickerState } from '../components/OnCanvasNodePicker.tsx'
import {
createOrUpdateConnectionBinding,
getConnectionBindings,
} from '../connection/ConnectionBindingUtil'
import { getNextConnectionIndex } from '../connection/keepConnectionsAtBottom'
import {
DEFAULT_NODE_SPACING_PX,
NODE_HEADER_HEIGHT_PX,
NODE_ROW_HEADER_GAP_PX,
NODE_ROW_HEIGHT_PX,
PortDataType,
} from '../constants.tsx'
import { getNodePortConnections, getNodePorts } from '../nodes/nodePorts'
import { PortId } from '../ports/Port'
import { findFirstCompatiblePort } from './portCompatibility'
interface PointingPortInfo {
shapeId: TLShapeId
portId: PortId
terminal: 'start' | 'end'
dataType: PortDataType
}
export class PointingPort extends StateNode {
static override id = 'pointing_port'
info?: PointingPortInfo
override onEnter(info: PointingPortInfo): void {
this.info = info
}
override onPointerMove(info: TLPointerEventInfo): void {
if (this.editor.inputs.getIsDragging()) {
const allowsMultipleConnections = this.info?.terminal === 'start'
const hasExistingConnection = getNodePortConnections(this.editor, this.info!.shapeId).find(
(c) => c.ownPortId === this.info!.portId
)
if (!allowsMultipleConnections && hasExistingConnection) {
this.parent.transition('dragging_handle', {
...info,
target: 'handle',
shape: this.editor.getShape(hasExistingConnection.connectionId)!,
handle: this.editor
.getShapeHandles(hasExistingConnection.connectionId)!
.find((h) => h.id === this.info!.terminal),
})
return
}
const creatingMarkId = this.editor.markHistoryStoppingPoint()
const connectionShapeId = createShapeId()
const connectingTerminal = this.info!.terminal
const draggingTerminal = connectingTerminal === 'start' ? 'end' : 'start'
this.editor.createShape({
type: 'connection',
id: connectionShapeId,
x: this.editor.inputs.getCurrentPagePoint().x,
y: this.editor.inputs.getCurrentPagePoint().y,
index: getNextConnectionIndex(this.editor),
props: {
start: { x: 0, y: 0 },
end: { x: 0, y: 0 },
},
})
createOrUpdateConnectionBinding(this.editor, connectionShapeId, this.info!.shapeId, {
portId: this.info!.portId,
terminal: connectingTerminal,
})
const handle = this.editor
.getShapeHandles(connectionShapeId)
?.find((h) => h.id === draggingTerminal)
this.parent.transition('dragging_handle', {
...info,
target: 'handle',
shape: this.editor.getShape(connectionShapeId)!,
handle: handle!,
creatingMarkId,
isCreating: true,
})
}
}
override onPointerUp(info: TLPointerEventInfo): void {
this.onClick()
this.parent.transition('idle', info)
}
private onClick() {
if (this.info?.terminal !== 'start') return
const bounds = this.editor.getShapePageBounds(this.info!.shapeId)
if (!bounds) return
const targetPositionInPageSpace = {
x: bounds.right + DEFAULT_NODE_SPACING_PX,
y: bounds.top,
}
const connectionShapeId = createShapeId()
this.editor.createShape({
type: 'connection',
id: connectionShapeId,
x: bounds.right,
y: bounds.top,
index: getNextConnectionIndex(this.editor),
})
createOrUpdateConnectionBinding(this.editor, connectionShapeId, this.info!.shapeId, {
portId: this.info!.portId,
terminal: 'start',
})
const targetPositionInConnectionSpace = this.editor
.getPointInShapeSpace(connectionShapeId, targetPositionInPageSpace)
.addXY(0, NODE_HEADER_HEIGHT_PX + NODE_ROW_HEADER_GAP_PX + NODE_ROW_HEIGHT_PX / 2)
this.editor.updateShape({
id: connectionShapeId,
type: 'connection',
props: {
end: targetPositionInConnectionSpace.toJson(),
},
})
onCanvasNodePickerState.set(this.editor, {
connectionShapeId,
location: 'end',
onPick: (nodeType, terminalInPageSpace) => {
const newNodeId = createShapeId()
this.editor.createShape({
type: 'node',
id: newNodeId,
x: terminalInPageSpace.x,
y: terminalInPageSpace.y,
props: {
node: nodeType,
},
})
this.editor.select(newNodeId)
const ports = getNodePorts(this.editor, newNodeId)
const firstCompatibleInputPort = findFirstCompatiblePort(
Object.values(ports),
'end',
this.info!.dataType
)
if (firstCompatibleInputPort) {
this.editor.updateShape({
id: newNodeId,
type: 'node',
x: terminalInPageSpace.x - firstCompatibleInputPort.x,
y: terminalInPageSpace.y - firstCompatibleInputPort.y,
})
createOrUpdateConnectionBinding(this.editor, connectionShapeId, newNodeId, {
portId: firstCompatibleInputPort.id,
terminal: 'end',
})
}
},
onClose: () => {
const connection = this.editor.getShape(connectionShapeId)
if (!connection || !this.editor.isShapeOfType(connection, 'connection')) return
const bindings = getConnectionBindings(this.editor, connection)
if (!bindings.start || !bindings.end) {
this.editor.deleteShapes([connection.id])
}
},
})
}
}

View File

@@ -0,0 +1,93 @@
import classNames from 'classnames'
import { TLShapeId, useEditor, useValue, VecModel } from 'tldraw'
import { PORT_TYPE_COLORS, PortDataType } from '../constants'
import { getNodePorts } from '../nodes/nodePorts'
import { portState } from './portState'
export type PortId = string
/**
* Port ids are unique within a shape. To identify a port we need both the shape id and the port id.
*/
export interface PortIdentifier {
shapeId: TLShapeId
portId: PortId
}
/**
* Shapes define their ports with a position, id, whether they're the start (an output) or end
* (an input) of a connection, and the data type of data they carry.
*/
export interface ShapePort extends VecModel {
id: PortId
terminal: 'start' | 'end'
dataType: PortDataType
/** When true, this input port accepts multiple simultaneous connections. */
multi?: boolean
}
/**
* This react component renders a port with its data-type color.
*/
export function Port({ shapeId, portId }: { shapeId: TLShapeId; portId: PortId }) {
const editor = useEditor()
const port = useValue(
'port',
() => {
const shape = editor.getShape(shapeId)
if (!shape || !editor.isShapeOfType(shape, 'node')) return null
return getNodePorts(editor, shape)?.[portId]
},
[shapeId, portId, editor]
)
if (!port) throw new Error(`Port ${portId} not found on shape ${shapeId}`)
const isHinting = useValue(
'isHinting',
() => {
const { hintingPort } = portState.get(editor)
return hintingPort && hintingPort.portId === portId && hintingPort.shapeId === shapeId
},
[editor, shapeId, portId]
)
const isEligible = useValue(
'isEligible',
() => {
const { eligiblePorts } = portState.get(editor)
if (!eligiblePorts) return false
if (eligiblePorts.terminal !== port.terminal) return false
if (eligiblePorts.excludeNodes?.has(shapeId)) return false
// type compatibility: 'any' matches everything, otherwise types must match
if (
eligiblePorts.dataType &&
eligiblePorts.dataType !== 'any' &&
port.dataType !== 'any' &&
eligiblePorts.dataType !== port.dataType
)
return false
return true
},
[editor, shapeId, port.terminal, port.dataType]
)
const color = PORT_TYPE_COLORS[port.dataType]
return (
<div
className={classNames(
`Port Port_${port.terminal}`,
isHinting ? 'Port_hinting' : isEligible ? 'Port_eligible' : undefined
)}
style={{ '--port-color': color } as React.CSSProperties}
onPointerDown={() => {
editor.setCurrentTool('select.pointing_port', {
shapeId,
portId,
terminal: port.terminal,
dataType: port.dataType,
})
}}
/>
)
}

View File

@@ -0,0 +1,42 @@
import { Editor, Vec, VecLike } from 'tldraw'
import { getNodePortConnections, getNodePorts } from '../nodes/nodePorts'
import { ShapePort } from './Port'
export function getPortAtPoint(
editor: Editor,
point: VecLike,
opts?: { terminal?: 'start' | 'end'; margin?: number }
) {
const shape = editor.getShapeAtPoint(point, {
hitInside: true,
filter: (shape) => editor.isShapeOfType(shape, 'node'),
...opts,
})
if (!shape || !editor.isShapeOfType(shape, 'node')) return null
const ports = getNodePorts(editor, shape)
if (!ports) return null
const shapeTransform = editor.getShapePageTransform(shape)
let bestPort: ShapePort | null = null
let bestDistance = Infinity
for (const port of Object.values(ports)) {
if (opts?.terminal && port.terminal !== opts.terminal) continue
const portInPageSpace = shapeTransform.applyToPoint(port)
const distance = Vec.Dist(point, portInPageSpace)
if (distance < bestDistance) {
bestPort = port
bestDistance = distance
}
}
if (!bestPort) return null
const existingConnections = getNodePortConnections(editor, shape).filter(
(c) => c.ownPortId === bestPort.id
)
return { shape, port: bestPort, existingConnections }
}

View File

@@ -0,0 +1,22 @@
import { ShapePort } from './Port'
export function arePortDataTypesCompatible(a: ShapePort['dataType'], b: ShapePort['dataType']) {
return a === 'any' || b === 'any' || a === b
}
export function findFirstCompatiblePort(
ports: ShapePort[],
terminal: ShapePort['terminal'],
dataType: ShapePort['dataType']
) {
// Prefer an exact type match over an 'any' match so that e.g. a text
// source connects to the 'prompt' port rather than a generic 'any' port.
const candidates = ports.filter((port) => {
return port.terminal === terminal && arePortDataTypesCompatible(port.dataType, dataType)
})
return (
candidates.find((port) => port.dataType === dataType) ??
candidates.find((port) => port.dataType === 'any' || dataType === 'any') ??
candidates[0]
)
}

View File

@@ -0,0 +1,29 @@
import { Editor, TLShapeId } from 'tldraw'
import { PortDataType } from '../constants'
import { EditorAtom } from '../utils'
import { PortIdentifier } from './Port'
/**
* The UI state for ports. These mostly highlight ports relevant to the user's current action.
*/
export interface PortState {
hintingPort: PortIdentifier | null
eligiblePorts: {
terminal: 'start' | 'end'
excludeNodes: Set<TLShapeId> | null
/** Only ports matching this data type are eligible. */
dataType: PortDataType | null
} | null
}
export const portState = new EditorAtom<PortState>('port state', () => ({
hintingPort: null,
eligiblePorts: null,
}))
export function updatePortState(editor: Editor, update: Partial<PortState>) {
portState.update(editor, (state) => {
const newState = { ...state, ...update }
return newState
})
}

View File

@@ -0,0 +1,154 @@
import { createBindingId, createShapeId, Editor, TLShapeId, VecModel } from 'tldraw'
import { getNodePortConnections } from '../nodes/nodePorts'
import { NodeShape } from '../nodes/NodeShapeUtil'
import { NodeType } from '../nodes/nodeTypes'
import {
PipelineTemplate,
saveTemplate,
SerializedTemplateConnection,
SerializedTemplateNode,
} from './templateState'
/**
* Save the currently selected nodes and their inter-connections as a
* reusable template.
*/
export function saveSelectionAsTemplate(
editor: Editor,
name: string,
description: string
): PipelineTemplate | null {
const selectedShapes = editor.getSelectedShapes()
const nodeShapes = selectedShapes.filter((s): s is NodeShape => s.type === 'node')
if (nodeShapes.length < 2) return null
// Compute bounding box for relative positioning.
let minX = Infinity
let minY = Infinity
for (const s of nodeShapes) {
minX = Math.min(minX, s.x)
minY = Math.min(minY, s.y)
}
const selectedIds = new Set<TLShapeId>(nodeShapes.map((s) => s.id))
// Map shape IDs to local IDs.
const idMap = new Map<TLShapeId, string>()
const nodes: SerializedTemplateNode[] = nodeShapes.map((s, i) => {
const localId = `n${i}`
idMap.set(s.id, localId)
return {
localId,
nodeType: s.props.node as NodeType,
relativeX: s.x - minX,
relativeY: s.y - minY,
}
})
// Collect connections that are entirely within the selection.
const connections: SerializedTemplateConnection[] = []
const seenConnections = new Set<string>()
for (const shape of nodeShapes) {
const portConns = getNodePortConnections(editor, shape)
for (const conn of portConns) {
if (!selectedIds.has(conn.connectedShapeId)) continue
// Only record from the 'start' terminal to avoid duplication.
if (conn.terminal !== 'start') continue
const key = conn.connectionId
if (seenConnections.has(key)) continue
seenConnections.add(key)
const fromLocal = idMap.get(shape.id)
const toLocal = idMap.get(conn.connectedShapeId)
if (!fromLocal || !toLocal) continue
connections.push({
fromLocalId: fromLocal,
fromPortId: conn.ownPortId,
toLocalId: toLocal,
toPortId: conn.connectedPortId,
})
}
}
const template: PipelineTemplate = {
id: `tpl_${Date.now()}_${Math.random().toString(36).slice(2, 8)}`,
name,
description,
createdAt: Date.now(),
nodes,
connections,
}
saveTemplate(template)
return template
}
/**
* Stamp a saved template onto the canvas at the given position.
*/
export function stampTemplate(editor: Editor, template: PipelineTemplate, position: VecModel) {
editor.markHistoryStoppingPoint('stamp template')
// Create shapes.
const localIdToShapeId = new Map<string, TLShapeId>()
editor.run(() => {
for (const tNode of template.nodes) {
const shapeId = createShapeId()
localIdToShapeId.set(tNode.localId, shapeId)
editor.createShape({
id: shapeId,
type: 'node',
x: position.x + tNode.relativeX,
y: position.y + tNode.relativeY,
props: { node: tNode.nodeType },
})
}
// Create connections.
for (const conn of template.connections) {
const fromId = localIdToShapeId.get(conn.fromLocalId)
const toId = localIdToShapeId.get(conn.toLocalId)
if (!fromId || !toId) continue
const connectionId = createShapeId()
editor.createShape({
id: connectionId,
type: 'connection',
props: {
start: { x: 0, y: 0 },
end: { x: 100, y: 0 },
},
})
editor.createBinding({
id: createBindingId(),
type: 'connection',
fromId: connectionId,
toId: fromId,
props: {
terminal: 'start',
portId: conn.fromPortId,
},
})
editor.createBinding({
id: createBindingId(),
type: 'connection',
fromId: connectionId,
toId: toId,
props: {
terminal: 'end',
portId: conn.toPortId,
},
})
}
// Select all the new shapes.
editor.select(...Array.from(localIdToShapeId.values()))
})
}

View File

@@ -0,0 +1,57 @@
import { NodeType } from '../nodes/nodeTypes'
/**
* A saved pipeline template — a serialized subgraph that can be stamped
* onto the canvas as a reusable "design machine".
*/
export interface PipelineTemplate {
id: string
name: string
description: string
createdAt: number
/** Serialized node shapes (relative positions). */
nodes: SerializedTemplateNode[]
/** Serialized connections between the nodes. */
connections: SerializedTemplateConnection[]
}
export interface SerializedTemplateNode {
/** A local ID used only within this template for wiring connections. */
localId: string
nodeType: NodeType
/** Position relative to the top-left corner of the template's bounding box. */
relativeX: number
relativeY: number
}
export interface SerializedTemplateConnection {
fromLocalId: string
fromPortId: string
toLocalId: string
toPortId: string
}
const STORAGE_KEY = 'image-pipeline-templates'
/** Load all saved templates from localStorage. */
export function loadTemplates(): PipelineTemplate[] {
try {
const raw = localStorage.getItem(STORAGE_KEY)
return raw ? JSON.parse(raw) : []
} catch {
return []
}
}
/** Save a template to localStorage. */
export function saveTemplate(template: PipelineTemplate) {
const templates = loadTemplates()
templates.push(template)
localStorage.setItem(STORAGE_KEY, JSON.stringify(templates))
}
/** Delete a template by id. */
export function deleteTemplate(id: string) {
const templates = loadTemplates().filter((t) => t.id !== id)
localStorage.setItem(STORAGE_KEY, JSON.stringify(templates))
}

View File

@@ -0,0 +1,29 @@
import { atom, Atom, Editor, WeakCache } from 'tldraw'
/**
* EditorAtom is an atom (tldraw's reactive signal) that is scoped to a specific editor. It's useful
* for storing and reacting to state that's shared between several components.
*/
export class EditorAtom<T> {
private states = new WeakCache<Editor, Atom<T>>()
constructor(
private name: string,
private getInitialState: (editor: Editor) => T
) {}
getAtom(editor: Editor) {
return this.states.get(editor, () => atom(this.name, this.getInitialState(editor)))
}
get(editor: Editor) {
return this.getAtom(editor).get()
}
update(editor: Editor, update: (state: T) => T) {
return this.getAtom(editor).update(update)
}
set(editor: Editor, state: T) {
return this.getAtom(editor).set(state)
}
}

View File

@@ -0,0 +1,8 @@
/**
* Delays execution for a specified number of milliseconds.
* @param ms - The number of milliseconds to delay.
* @returns A promise that resolves after the specified delay.
*/
export function sleep(ms: number): Promise<void> {
return new Promise((resolve) => setTimeout(resolve, ms))
}

View File

@@ -0,0 +1,22 @@
{
"compilerOptions": {
"target": "ES2020",
"useDefineForClassFields": true,
"lib": ["DOM", "DOM.Iterable", "ESNext"],
"module": "ESNext",
"skipLibCheck": true,
"moduleResolution": "bundler",
"allowImportingTsExtensions": true,
"resolveJsonModule": true,
"isolatedModules": true,
"noEmit": true,
"jsx": "react-jsx",
"strict": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"noFallthroughCasesInSwitch": true,
"types": ["vite/client", "./worker-configuration.d.ts"]
},
"include": ["src", "worker", "vite.config.ts"],
"references": [{ "path": "../../packages/tldraw" }]
}

View File

@@ -0,0 +1,8 @@
import { cloudflare } from '@cloudflare/vite-plugin'
import react from '@vitejs/plugin-react'
import { defineConfig } from 'vite'
// https://vitejs.dev/config/
export default defineConfig({
plugins: [cloudflare(), react()],
})

View File

@@ -0,0 +1,18 @@
import { replicate } from './replicate'
import type { ImageProvider } from './types'
export type {
GenerateParams,
GenerateResult,
ImageProvider,
UpscaleParams,
UpscaleResult,
} from './types'
export function getProvider(_name: string): ImageProvider {
return replicate
}
export function getUpscaleProvider(_method: string): ImageProvider {
return replicate
}

View File

@@ -0,0 +1,235 @@
import { resolveImage } from './types'
import type {
GenerateParams,
GenerateResult,
ImageProvider,
UpscaleParams,
UpscaleResult,
} from './types'
/** Maps ControlNet mode IDs to Flux ControlNet model identifiers. */
const CONTROLNET_MODELS: Record<string, string> = {
canny: 'black-forest-labs/flux-canny-dev',
depth: 'black-forest-labs/flux-depth-dev',
// No Flux-native pose/segmentation models — fall back to canny
pose: 'black-forest-labs/flux-canny-dev',
segmentation: 'black-forest-labs/flux-depth-dev',
}
export const replicate: ImageProvider = {
name: 'replicate',
async generate(params: GenerateParams, env: Env): Promise<GenerateResult> {
const apiToken = env.REPLICATE_API_TOKEN
if (!apiToken) {
throw new Error('REPLICATE_API_TOKEN is not configured')
}
// Use a ControlNet model when ControlNet params are present
if (params.controlNetMode && params.referenceImageUrl) {
return generateWithControlNet(params, apiToken, env)
}
// Google models (Nano Banana, Imagen)
if (
params.modelId === 'nano-banana-pro' ||
params.modelId === 'nano-banana' ||
params.modelId === 'imagen-4-fast'
) {
return generateWithGoogle(params, apiToken)
}
return generateWithFlux(params, apiToken)
},
async upscale(params: UpscaleParams, env: Env): Promise<UpscaleResult> {
const apiToken = env.REPLICATE_API_TOKEN
if (!apiToken) {
throw new Error('REPLICATE_API_TOKEN is not configured')
}
const response = await fetch(
'https://api.replicate.com/v1/models/nightmareai/real-esrgan/predictions',
{
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${apiToken}`,
Prefer: 'wait',
},
body: JSON.stringify({
input: {
image: params.imageUrl,
scale: params.scale,
},
}),
}
)
if (!response.ok) {
const text = await response.text()
throw new Error(`Replicate upscale error ${response.status}: ${text}`)
}
const data = (await response.json()) as { output: string }
return { imageUrl: data.output }
},
}
async function generateWithFlux(params: GenerateParams, apiToken: string): Promise<GenerateResult> {
const replicateModel =
params.modelId === 'flux-schnell'
? 'black-forest-labs/flux-schnell'
: params.modelId === 'flux-pro'
? 'black-forest-labs/flux-1.1-pro'
: 'black-forest-labs/flux-dev'
const response = await fetch(
`https://api.replicate.com/v1/models/${replicateModel}/predictions`,
{
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${apiToken}`,
Prefer: 'wait',
},
body: JSON.stringify({
input: {
prompt: params.prompt,
num_inference_steps: params.steps ?? 20,
guidance: params.cfgScale ?? 7,
seed: params.seed ?? null,
aspect_ratio: '1:1',
// Flux Pro uses safety_tolerance (1=strictest), others use disable_safety_checker
...(params.modelId === 'flux-pro'
? { safety_tolerance: 1 }
: { disable_safety_checker: false }),
...(params.referenceImageUrl ? { image: params.referenceImageUrl } : {}),
},
}),
}
)
if (!response.ok) {
const text = await response.text()
throw new Error(`Replicate error ${response.status}: ${text}`)
}
const data = (await response.json()) as {
output: string | string[]
seed?: number
}
const imageUrl = Array.isArray(data.output) ? data.output[0] : data.output
return {
imageUrl,
seed: data.seed ?? params.seed ?? 0,
}
}
/**
* Use a Google model on Replicate (Nano Banana, Imagen).
*/
async function generateWithGoogle(
params: GenerateParams,
apiToken: string
): Promise<GenerateResult> {
const modelMap: Record<string, string> = {
'nano-banana-pro': 'google/nano-banana-pro',
'nano-banana': 'google/nano-banana',
'imagen-4-fast': 'google/imagen-4-fast',
}
const replicateModel = modelMap[params.modelId] ?? 'google/nano-banana-pro'
const response = await fetch(
`https://api.replicate.com/v1/models/${replicateModel}/predictions`,
{
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${apiToken}`,
Prefer: 'wait',
},
body: JSON.stringify({
input: {
prompt: params.prompt,
aspect_ratio: '1:1',
...(params.referenceImageUrl ? { image_input: [params.referenceImageUrl] } : {}),
},
}),
}
)
if (!response.ok) {
const text = await response.text()
throw new Error(`Replicate Google model error ${response.status}: ${text}`)
}
const data = (await response.json()) as {
output: string | string[]
seed?: number
}
const imageUrl = Array.isArray(data.output) ? data.output[0] : data.output
return {
imageUrl,
seed: data.seed ?? params.seed ?? 0,
}
}
/**
* Use a Flux ControlNet model on Replicate.
* The reference image is passed as the control image, and the mode
* selects the appropriate variant (canny or depth).
*/
async function generateWithControlNet(
params: GenerateParams,
apiToken: string,
env: Env
): Promise<GenerateResult> {
const model = CONTROLNET_MODELS[params.controlNetMode!] ?? CONTROLNET_MODELS.canny
// Replicate accepts https URLs and data URIs directly.
// Only resolve R2 paths that Replicate can't access.
let imageInput = params.referenceImageUrl!
if (imageInput.startsWith('/api/images/')) {
const { dataUrl } = await resolveImage(imageInput, env)
imageInput = dataUrl
}
const response = await fetch(`https://api.replicate.com/v1/models/${model}/predictions`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${apiToken}`,
Prefer: 'wait',
},
body: JSON.stringify({
input: {
control_image: imageInput,
prompt: params.prompt,
num_inference_steps: params.steps ?? 28,
guidance: params.cfgScale ?? 30,
...(params.seed != null ? { seed: params.seed } : {}),
output_format: 'png',
disable_safety_checker: false,
},
}),
})
if (!response.ok) {
const text = await response.text()
throw new Error(`Replicate ControlNet error ${response.status}: ${text}`)
}
const data = (await response.json()) as {
output: string | string[]
seed?: number
}
const imageUrl = Array.isArray(data.output) ? data.output[0] : data.output
return {
imageUrl,
seed: data.seed ?? params.seed ?? 0,
}
}

View File

@@ -0,0 +1,95 @@
export interface GenerateParams {
modelId: string
prompt: string
negativePrompt?: string
steps: number
cfgScale: number
seed: number | null
controlNetMode?: string
controlNetStrength?: number
referenceImageUrl?: string
}
export interface GenerateResult {
imageUrl: string
seed: number
}
export interface UpscaleParams {
imageUrl: string
scale: number
method: string
}
export interface UpscaleResult {
imageUrl: string
}
export interface ImageProvider {
name: string
generate(params: GenerateParams, env: Env): Promise<GenerateResult>
upscale?(params: UpscaleParams, env: Env): Promise<UpscaleResult>
}
/**
* Resolve an image URL (data URL, R2 path, or external URL) into a Blob
* and a data URL that external APIs can consume.
*/
export async function resolveImage(
url: string,
env: Env
): Promise<{ blob: Blob; dataUrl: string }> {
if (url.startsWith('data:')) {
const [header, data] = url.split(',')
const mimeMatch = header.match(/data:([^;]+)/)
const mime = mimeMatch?.[1] ?? 'image/png'
if (header.includes('base64')) {
const binary = atob(data)
const bytes = new Uint8Array(binary.length)
for (let i = 0; i < binary.length; i++) {
bytes[i] = binary.charCodeAt(i)
}
return { blob: new Blob([bytes], { type: mime }), dataUrl: url }
}
const decoded = decodeURIComponent(data)
const bytes = new TextEncoder().encode(decoded)
const b64 = arrayBufferToBase64(bytes.buffer as ArrayBuffer)
return {
blob: new Blob([bytes], { type: mime }),
dataUrl: `data:${mime};base64,${b64}`,
}
}
// R2 path — read directly from the bucket
if (url.startsWith('/api/images/') && env.IMAGE_BUCKET) {
const imageId = url.slice('/api/images/'.length)
const object = await env.IMAGE_BUCKET.get(imageId)
if (!object) throw new Error(`Image not found in R2: ${imageId}`)
const buf = await object.arrayBuffer()
const mime = object.httpMetadata?.contentType ?? 'image/png'
return {
blob: new Blob([buf], { type: mime }),
dataUrl: `data:${mime};base64,${arrayBufferToBase64(buf)}`,
}
}
// External URL
const res = await fetch(url)
if (!res.ok) throw new Error(`Failed to fetch image: ${res.status}`)
const buf = await res.arrayBuffer()
const mime = res.headers.get('content-type') ?? 'image/png'
return {
blob: new Blob([buf], { type: mime }),
dataUrl: `data:${mime};base64,${arrayBufferToBase64(buf)}`,
}
}
function arrayBufferToBase64(buffer: ArrayBuffer): string {
const bytes = new Uint8Array(buffer)
const chunks: string[] = []
const chunkSize = 8192
for (let i = 0; i < bytes.length; i += chunkSize) {
chunks.push(String.fromCharCode(...bytes.subarray(i, i + chunkSize)))
}
return btoa(chunks.join(''))
}

View File

@@ -0,0 +1,103 @@
import { IRequest } from 'itty-router'
import { getProvider } from '../providers'
import type { GenerateParams } from '../providers'
/**
* Request body for the /api/generate endpoint.
*/
interface GenerateRequest {
/** The provider:model string, e.g. "stable-diffusion:sdxl" */
model: string
/** The text prompt describing the desired image */
prompt: string
/** Optional negative prompt */
negativePrompt?: string
/** Number of inference steps (default: 20) */
steps?: number
/** Classifier-free guidance scale (default: 7) */
cfgScale?: number
/** Seed for reproducibility (default: random) */
seed?: number
/** ControlNet mode if applicable */
controlNetMode?: string
/** ControlNet strength (0-100) */
controlNetStrength?: number
/** Reference image URL for ControlNet */
referenceImageUrl?: string
}
/**
* POST /api/generate
*
* Calls an AI image generation provider and returns the generated image.
* Supports multiple providers via the model string format "provider:model".
*
* Returns: { imageUrl: string, seed: number }
*/
export async function handleGenerate(request: IRequest, env: Env) {
const body = (await request.json()) as GenerateRequest
if (!body.prompt) {
return new Response(JSON.stringify({ error: 'prompt is required' }), {
status: 400,
headers: { 'Content-Type': 'application/json' },
})
}
const [providerName, modelId] = (body.model ?? 'flux:flux-dev').split(':')
try {
const provider = getProvider(providerName)
const params: GenerateParams = {
modelId: modelId ?? '',
prompt: body.prompt,
negativePrompt: body.negativePrompt,
steps: body.steps ?? 20,
cfgScale: body.cfgScale ?? 7,
seed: body.seed ?? null,
controlNetMode: body.controlNetMode,
controlNetStrength: body.controlNetStrength,
referenceImageUrl: body.referenceImageUrl,
}
let result = await provider.generate(params, env)
// Optionally persist the image to R2.
if (env.IMAGE_BUCKET && result.imageUrl?.startsWith('data:')) {
const imageId = `gen_${Date.now()}_${Math.random().toString(36).slice(2, 8)}`
const blob = dataUrlToBlob(result.imageUrl)
await env.IMAGE_BUCKET.put(imageId, blob, {
httpMetadata: { contentType: 'image/png' },
})
result = { ...result, imageUrl: `/api/images/${imageId}` }
}
return new Response(JSON.stringify(result), {
headers: { 'Content-Type': 'application/json' },
})
} catch (e: any) {
console.error('Generate error:', e)
return new Response(JSON.stringify({ error: e.message ?? 'Generation failed' }), {
status: 500,
headers: { 'Content-Type': 'application/json' },
})
}
}
// ---------------------------------------------------------------------------
// Utilities
// ---------------------------------------------------------------------------
function dataUrlToBlob(dataUrl: string): ArrayBuffer {
const [header, base64] = dataUrl.split(',')
if (header.includes('base64')) {
const binary = atob(base64)
const bytes = new Uint8Array(binary.length)
for (let i = 0; i < binary.length; i++) {
bytes[i] = binary.charCodeAt(i)
}
return bytes.buffer as ArrayBuffer
}
// For SVG data URLs, just encode as UTF-8
return new TextEncoder().encode(decodeURIComponent(base64)).buffer as ArrayBuffer
}

View File

@@ -0,0 +1,116 @@
import { IRequest } from 'itty-router'
import { resolveImage } from '../providers/types'
interface GenerateTextRequest {
input?: string
prompt: string
}
/**
* POST /api/generate-text
*
* Takes an optional input (image or text) and a prompt, then calls
* google/gemini-3.5-flash on Replicate to generate text.
* Falls back to a placeholder if no API token is configured.
*/
export async function handleGenerateText(request: IRequest, env: Env) {
const body = (await request.json()) as GenerateTextRequest
if (!body.prompt) {
return new Response(JSON.stringify({ error: 'prompt is required' }), {
status: 400,
headers: { 'Content-Type': 'application/json' },
})
}
const apiToken = env.REPLICATE_API_TOKEN
if (!apiToken) {
return new Response(JSON.stringify(generateTextPlaceholder(body)), {
headers: { 'Content-Type': 'application/json' },
})
}
// Coerce input to string so downstream .startsWith() never crashes
const inputStr = body.input != null ? String(body.input) : null
// Detect whether the input is an image or text
const isImage =
inputStr != null &&
(inputStr.startsWith('data:image/') ||
inputStr.startsWith('/api/images/') ||
inputStr.startsWith('https://') ||
inputStr.startsWith('http://'))
try {
// Build the prompt — if input is text, prepend it to the prompt
let prompt = body.prompt
if (inputStr && !isImage) {
prompt = `Context:\n${inputStr}\n\n${body.prompt}`
}
// Build the images array if we have an image input
const images: string[] = []
if (inputStr && isImage) {
const { dataUrl } = await resolveImage(inputStr, env)
images.push(dataUrl)
}
const input: Record<string, unknown> = {
prompt,
max_output_tokens: 1024,
}
if (images.length > 0) {
input.images = images
}
const response = await fetch(
'https://api.replicate.com/v1/models/google/gemini-3.5-flash/predictions',
{
method: 'POST',
headers: {
'Content-Type': 'application/json',
Authorization: `Bearer ${apiToken}`,
Prefer: 'wait',
},
body: JSON.stringify({ input }),
}
)
if (!response.ok) {
const err = await response.text()
throw new Error(`Replicate error: ${response.status} ${err}`)
}
const result = (await response.json()) as { output?: string | string[] }
const output = Array.isArray(result.output) ? result.output.join('') : result.output
if (!output) throw new Error('No output from text generation')
return new Response(JSON.stringify({ text: output }), {
headers: { 'Content-Type': 'application/json' },
})
} catch (e: any) {
console.error('Generate text error:', e)
return new Response(JSON.stringify({ error: e.message ?? 'Text generation failed' }), {
status: 500,
headers: { 'Content-Type': 'application/json' },
})
}
}
function generateTextPlaceholder(body: GenerateTextRequest): { text: string } {
const inputStr = body.input != null ? String(body.input) : null
const isImage =
inputStr != null &&
(inputStr.startsWith('data:image/') ||
inputStr.startsWith('/api/images/') ||
inputStr.startsWith('https://') ||
inputStr.startsWith('http://'))
const inputDesc = inputStr
? isImage
? '[image provided]'
: `[text: "${inputStr.slice(0, 40)}${inputStr.length > 40 ? '...' : ''}"]`
: '[no input]'
return {
text: `[Placeholder] Prompt: "${body.prompt.slice(0, 60)}${body.prompt.length > 60 ? '...' : ''}" | Input: ${inputDesc} — Set REPLICATE_API_TOKEN for real text generation.`,
}
}

View File

@@ -0,0 +1,68 @@
import { IRequest } from 'itty-router'
/**
* POST /api/images/:imageId
*
* Upload a generated image to R2 for persistence.
*/
export async function handleImageUpload(request: IRequest, env: Env) {
const { imageId } = request.params
const contentType = request.headers.get('content-type') ?? 'image/png'
if (!contentType.startsWith('image/')) {
return new Response(JSON.stringify({ error: 'Invalid content type' }), {
status: 400,
headers: { 'Content-Type': 'application/json' },
})
}
// Don't overwrite existing images.
if (await env.IMAGE_BUCKET.head(imageId)) {
return new Response(JSON.stringify({ ok: true }), {
headers: { 'Content-Type': 'application/json' },
})
}
await env.IMAGE_BUCKET.put(imageId, request.body, {
httpMetadata: { contentType },
})
return new Response(JSON.stringify({ ok: true }), {
headers: { 'Content-Type': 'application/json' },
})
}
/**
* GET /api/images/:imageId
*
* Download a generated image from R2 with caching.
*/
export async function handleImageDownload(request: IRequest, env: Env, ctx: ExecutionContext) {
const { imageId } = request.params
// Check edge cache first.
// Use caches.open() to avoid DOM/Workers CacheStorage type conflict.
const cache = await caches.open('images')
const cacheKey = new Request(request.url, { headers: request.headers })
const cachedResponse = await cache.match(cacheKey)
if (cachedResponse) {
return cachedResponse
}
const object = await env.IMAGE_BUCKET.get(imageId)
if (!object) {
return new Response('Not found', { status: 404 })
}
const headers = new Headers()
headers.set('content-type', object.httpMetadata?.contentType ?? 'image/png')
// Generated images are immutable — cache them aggressively.
headers.set('cache-control', 'public, max-age=31536000, immutable')
const response = new Response(object.body, { headers })
// Cache in the edge for future requests.
ctx.waitUntil(cache.put(cacheKey, response.clone()))
return response
}

View File

@@ -0,0 +1,101 @@
import { IRequest } from 'itty-router'
import { resolveImage } from '../providers/types'
interface IPAdapterRequest {
imageUrl: string
prompt: string
scale: number
steps: number
}
/**
* POST /api/ip-adapter
*
* Generates an image guided by a reference image and text prompt using
* IP-Adapter SDXL on Replicate. Falls back to a placeholder if no API key.
*/
export async function handleIPAdapter(request: IRequest, env: Env) {
const body = (await request.json()) as IPAdapterRequest
if (!body.imageUrl) {
return new Response(JSON.stringify({ error: 'imageUrl is required' }), {
status: 400,
headers: { 'Content-Type': 'application/json' },
})
}
const apiKey = env.REPLICATE_API_TOKEN
if (!apiKey) {
return new Response(JSON.stringify(ipAdapterPlaceholder(body)), {
headers: { 'Content-Type': 'application/json' },
})
}
try {
const { dataUrl } = await resolveImage(body.imageUrl, env)
const prediction = await fetch('https://api.replicate.com/v1/predictions', {
method: 'POST',
headers: {
Authorization: `Bearer ${apiKey}`,
'Content-Type': 'application/json',
Prefer: 'wait',
},
body: JSON.stringify({
version: '904dc004af1dba5c9b13fc9e41635aeb2f9a177896a396ab3393f3f6493dbdd4',
input: {
image: dataUrl,
prompt: body.prompt || 'best quality, high quality',
scale: body.scale ?? 0.6,
num_inference_steps: body.steps ?? 30,
},
}),
})
if (!prediction.ok) {
const err = await prediction.text()
throw new Error(`Replicate error: ${prediction.status} ${err}`)
}
const result = (await prediction.json()) as { output?: string[] | string }
const outputUrl = Array.isArray(result.output) ? result.output[0] : result.output
if (!outputUrl) throw new Error('No output from IP-Adapter')
// Persist to R2 if available
let imageUrl = outputUrl
if (env.IMAGE_BUCKET) {
const imageRes = await fetch(outputUrl)
const imageData = await imageRes.arrayBuffer()
const id = crypto.randomUUID()
await env.IMAGE_BUCKET.put(id, imageData, {
httpMetadata: { contentType: imageRes.headers.get('content-type') ?? 'image/png' },
})
imageUrl = `/api/images/${id}`
}
return new Response(JSON.stringify({ imageUrl }), {
headers: { 'Content-Type': 'application/json' },
})
} catch (e: any) {
console.error('IP-Adapter error:', e)
return new Response(JSON.stringify({ error: e.message ?? 'IP-Adapter failed' }), {
status: 500,
headers: { 'Content-Type': 'application/json' },
})
}
}
function ipAdapterPlaceholder(params: IPAdapterRequest) {
const hue = Math.floor(Math.random() * 360)
const prompt = (params.prompt || 'IP-Adapter').slice(0, 30)
const svg = `<svg xmlns="http://www.w3.org/2000/svg" width="1024" height="1024">
<defs><linearGradient id="bg" x1="0" y1="0" x2="1" y2="1">
<stop offset="0%" stop-color="hsl(${hue},50%,40%)"/>
<stop offset="100%" stop-color="hsl(${(hue + 80) % 360},45%,55%)"/>
</linearGradient></defs>
<rect width="1024" height="1024" fill="url(#bg)"/>
<text x="512" y="490" text-anchor="middle" fill="rgba(255,255,255,0.7)" font-family="sans-serif" font-size="22">${prompt}</text>
<text x="512" y="530" text-anchor="middle" fill="rgba(255,255,255,0.4)" font-family="sans-serif" font-size="14">IP-Adapter · scale ${params.scale} · placeholder</text>
</svg>`
return { imageUrl: `data:image/svg+xml,${encodeURIComponent(svg)}` }
}

View File

@@ -0,0 +1,119 @@
import { IRequest } from 'itty-router'
import { resolveImage } from '../providers/types'
interface StyleTransferRequest {
styleImageUrl: string
contentImageUrl?: string
prompt?: string
model: string
strength: number
}
/**
* POST /api/style-transfer
*
* Transfers the style of one image onto another (or generates a new image
* in that style) using fofr/style-transfer on Replicate.
* Falls back to a placeholder if no API key.
*/
export async function handleStyleTransfer(request: IRequest, env: Env) {
const body = (await request.json()) as StyleTransferRequest
if (!body.styleImageUrl) {
return new Response(JSON.stringify({ error: 'styleImageUrl is required' }), {
status: 400,
headers: { 'Content-Type': 'application/json' },
})
}
const apiKey = env.REPLICATE_API_TOKEN
if (!apiKey) {
return new Response(JSON.stringify(styleTransferPlaceholder(body)), {
headers: { 'Content-Type': 'application/json' },
})
}
try {
const { dataUrl: styleDataUrl } = await resolveImage(body.styleImageUrl, env)
const input: Record<string, unknown> = {
style_image: styleDataUrl,
prompt: body.prompt || '',
style_strength: body.strength ?? 0.5,
}
if (body.contentImageUrl) {
const { dataUrl: contentDataUrl } = await resolveImage(body.contentImageUrl, env)
input.structure_image = contentDataUrl
}
// Map model variant to the Replicate model parameter
const modelMap: Record<string, string> = {
fast: 'fast',
'high-quality': 'high-quality',
realistic: 'realistic',
cinematic: 'cinematic',
animated: 'animated',
}
input.model = modelMap[body.model] ?? 'fast'
const prediction = await fetch('https://api.replicate.com/v1/predictions', {
method: 'POST',
headers: {
Authorization: `Bearer ${apiKey}`,
'Content-Type': 'application/json',
Prefer: 'wait',
},
body: JSON.stringify({
version: 'f1023890703bc0a5a3a2c21b5e498833be5f6ef6e70e9daf6b9b3a4fd8309cf0',
input,
}),
})
if (!prediction.ok) {
const err = await prediction.text()
throw new Error(`Replicate error: ${prediction.status} ${err}`)
}
const result = (await prediction.json()) as { output?: string[] | string }
const outputUrl = Array.isArray(result.output) ? result.output[0] : result.output
if (!outputUrl) throw new Error('No output from style transfer')
// Persist to R2 if available
let imageUrl = outputUrl
if (env.IMAGE_BUCKET) {
const imageRes = await fetch(outputUrl)
const imageData = await imageRes.arrayBuffer()
const id = crypto.randomUUID()
await env.IMAGE_BUCKET.put(id, imageData, {
httpMetadata: { contentType: imageRes.headers.get('content-type') ?? 'image/png' },
})
imageUrl = `/api/images/${id}`
}
return new Response(JSON.stringify({ imageUrl }), {
headers: { 'Content-Type': 'application/json' },
})
} catch (e: any) {
console.error('Style transfer error:', e)
return new Response(JSON.stringify({ error: e.message ?? 'Style transfer failed' }), {
status: 500,
headers: { 'Content-Type': 'application/json' },
})
}
}
function styleTransferPlaceholder(params: StyleTransferRequest) {
const hue = Math.floor(Math.random() * 360)
const model = params.model || 'fast'
const svg = `<svg xmlns="http://www.w3.org/2000/svg" width="1024" height="1024">
<defs><linearGradient id="bg" x1="0" y1="0" x2="1" y2="1">
<stop offset="0%" stop-color="hsl(${hue},55%,35%)"/>
<stop offset="100%" stop-color="hsl(${(hue + 120) % 360},50%,50%)"/>
</linearGradient></defs>
<rect width="1024" height="1024" fill="url(#bg)"/>
<text x="512" y="490" text-anchor="middle" fill="rgba(255,255,255,0.7)" font-family="sans-serif" font-size="22">Style Transfer</text>
<text x="512" y="530" text-anchor="middle" fill="rgba(255,255,255,0.4)" font-family="sans-serif" font-size="14">${model} · strength ${params.strength} · placeholder</text>
</svg>`
return { imageUrl: `data:image/svg+xml,${encodeURIComponent(svg)}` }
}

View File

@@ -0,0 +1,57 @@
import { IRequest } from 'itty-router'
import { getUpscaleProvider } from '../providers'
import type { UpscaleParams } from '../providers'
interface UpscaleRequest {
/** URL of the image to upscale */
imageUrl: string
/** Scale factor (2 or 4) */
scale: number
/** Upscale method */
method: string
}
/**
* POST /api/upscale
*
* Upscales an image using an AI upscaler. Falls back to a placeholder
* if no API key is configured.
*/
export async function handleUpscale(request: IRequest, env: Env) {
const body = (await request.json()) as UpscaleRequest
if (!body.imageUrl) {
return new Response(JSON.stringify({ error: 'imageUrl is required' }), {
status: 400,
headers: { 'Content-Type': 'application/json' },
})
}
try {
const provider = getUpscaleProvider(body.method)
const params: UpscaleParams = {
imageUrl: body.imageUrl,
scale: body.scale,
method: body.method,
}
if (!provider.upscale) {
return new Response(
JSON.stringify({ error: `Provider "${provider.name}" does not support upscaling` }),
{ status: 400, headers: { 'Content-Type': 'application/json' } }
)
}
const result = await provider.upscale(params, env)
return new Response(JSON.stringify(result), {
headers: { 'Content-Type': 'application/json' },
})
} catch (e: any) {
console.error('Upscale error:', e)
return new Response(JSON.stringify({ error: e.message ?? 'Upscale failed' }), {
status: 500,
headers: { 'Content-Type': 'application/json' },
})
}
}

View File

@@ -0,0 +1,38 @@
import { AutoRouter, error, IRequest } from 'itty-router'
import { handleGenerate } from './routes/generate'
import { handleGenerateText } from './routes/generateText'
import { handleImageDownload, handleImageUpload } from './routes/images'
import { handleIPAdapter } from './routes/ipAdapter'
import { handleStyleTransfer } from './routes/styleTransfer'
import { handleUpscale } from './routes/upscale'
const router = AutoRouter<IRequest, [env: Env, ctx: ExecutionContext]>({
catch: (e) => {
console.error(e)
return error(e)
},
})
// Image generation endpoint — accepts prompt, model, and parameters
.post('/api/generate', handleGenerate)
// Upscale endpoint — accepts an image URL and scale factor
.post('/api/upscale', handleUpscale)
// IP-Adapter endpoint — reference image + prompt guided generation
.post('/api/ip-adapter', handleIPAdapter)
// Style transfer endpoint — transfers style between images
.post('/api/style-transfer', handleStyleTransfer)
// Text generation endpoint — multimodal AI text generation
.post('/api/generate-text', handleGenerateText)
// Upload/download generated images to/from R2 bucket
.post('/api/images/:imageId', handleImageUpload)
.get('/api/images/:imageId', handleImageDownload)
.all('*', () => new Response('Not found', { status: 404 }))
export default {
fetch: router.fetch,
}

View File

@@ -0,0 +1,29 @@
name = "image-pipeline-template"
main = "worker/worker.ts"
compatibility_date = "2025-05-08"
compatibility_flags = ["nodejs_compat"]
upload_source_maps = true
preview_urls = true
[[routes]]
pattern = "image-pipeline.templates.tldraw.dev"
custom_domain = true
[observability]
enabled = true
[assets]
not_found_handling = "single-page-application"
directory = "./dist/client"
# Store generated images in an R2 bucket so they persist across sessions.
[[r2_buckets]]
binding = 'IMAGE_BUCKET'
bucket_name = 'image-pipeline'
preview_bucket_name = 'image-pipeline-preview'
# Environment variables for AI provider API keys.
# Set these with: wrangler secret put REPLICATE_API_TOKEN
# Or use a .dev.vars file for local development:
# REPLICATE_API_TOKEN=...