Vector retrieve vs an agent that picks files

The problem

Website workflows pull a curated knowledge store from a scrape: App Profile, Persona, Audience, Brand Styles, crawl notes, logos, page stills, and user uploads. On the canvas that store is one knowledge node, not seventeen drag-and-drop files.

Automations still need the right context for the step they are running. A tip-card script wants different passages than an app-review presenter or an ad headline. Dumping every text file into the prompt (then truncating) ships fast and does not scale: early files win, later ones disappear, and name-dropping “use the App Profile” does nothing special unless something actually selects that file.

So the design question is: who is the librarian?

Three librarians

1. Deterministic metadata / rules (kind, tags, “primary only,” char caps). Cheap and debuggable. Fails when intent is not encoded in labels.
2. Agent as librarian (catalog of name, kind, tags, blurb → model picks files, then reads them).
3. Vector database (embed chunks, retrieve by similarity to the task / user prompt).

I already lived with (1) in spirit: link the store, concatenate text, cap characters. That taught me the product shape. The choice for the next step is between (2) and (3).

When an agent picker wins

An agent over a metadata catalog is a good fit when:

• the corpus is tiny (tens of files, not thousands of chunks)
• metadata is rich and stable (kind, tags, “use for script,” “use for overlay”)
• the job is routing, not recall (“which of these six notes inform the script?”)
• you want explainability (“I picked App Profile because…”)
• you have no embedding pipeline yet, and every scrape refresh would force a re-index

The costs show up every run:

• extra LLM latency and tokens just to choose files
• non-deterministic picks; hard to regression-test
• agents over-include, invent filenames, or ignore the catalog
• prompt injection via file names and descriptions

When vectors win

A vector index is a better librarian when:

• the corpus grows (crawl harvest, many notes, uploads)
• the query is semantic (“tone for a skeptical SMB buyer”), not “file named Audience”
• the same store serves many workflows with different questions
• you want bounded context by relevance score, not “everything linked, truncated”
• refresh mutates content often: re-embed beats re-prompting a picker with a stale catalog

Costs: embedding + storage, chunking quality bugs (“wrong paragraph, right file”), weaker precise ID lookup (“always use Site Logo”), and ops for reindex after refresh. Top-k can look random without metadata filters.

Tradeoff table

Why I am betting on vectors

For Content Shifted, knowledge is becoming a shared site brain, not a fixed six-file menu on the canvas.

1. The store grows. Scrape vs knowledge split, crawl harvest, and uploads push the catalog past what an agent should re-read on every automation run.
2. Workflows ask different questions of the same site. Tip posts, app reviews, and ads need different passages. One “picker prompt” does not generalize; similarity-to-task does.
3. Refresh already mutates files. App Profile and Brand Styles update after crawl. Re-embedding indexes cleanly. An agent still needs a fresh catalog summary every time.
4. Vectors also help an agent pick a file when I need that. An agentic step can query the index (“best screenshot for this script beat”) instead of scanning a long file list. The vector DB is the retrieval substrate; the agent stays the planner / tool user. I do not want an LLM as a slow, expensive vector database.

Short version: agents pick among labels; vectors recall among meanings. I chose meanings for the shared store, and keep agents for multi-step work that may call retrieve as a tool.

Hybrid shape (what I actually want)

Pure agent or pure vector is a strawman. The mature path is hybrid:

Crawl / refresh → knowledge files (+ metadata)
                    ↓
            embed + index (per account / project)
                    ↓
     automation runs → query = goal + user prompt
                    ↓
  filter by metadata (kind, origin, tags) → top-k chunks → prompt context

Optional agent path:
  agent decides strategy → same vector retrieve tool → attach passages / files

Metadata narrows the pool (text vs image, uploaded vs scraped, primary vs harvest). Vectors rank what text matters for this step. Agents optionally call that retrieve tool when the workflow is agentic, not when every strict transform needs a librarian LLM.

Pitfalls I am careful about

• Vectors do not replace slots. Logo images, tour video, and template layers still need explicit edges or roles. Retrieve is for prompt context, not for inventing wiring.
• Do not use an agent as a slow vector DB. If the only job is “find relevant passages,” embed and rank. Save the agent budget for tool loops and judgment.
• Eval over vibes. Golden tasks (“script should mention the pricing objection”) beat eyeballing top-k.

Pick the librarian that matches corpus size and query type. For a growing, shared knowledge store across many workflows, I bet on the vector index, and keep agent file-pick as a consumer of that index when needed.