6
Scaling the Analyst—From CLI to MCP-Connected Agents 17:54 Lena: We’ve built the dashboard and the semantic layer, but sometimes you don't want to open a browser at all. Sometimes you’re in the flow of development and you just want to hit the command line. How does the CLI fit into this data intelligence loop?
18:09 Miles: The CLI is the "power user" interface for the loop. Tools like the Databricks CLI or even custom-built MCP servers allow you to interact with your data models without any UI overhead. For instance, with the `databricks quality-monitors` command group, you can create, run, and list refreshes for your data health checks directly from your terminal. It’s about "headless" intelligence.
18:31 Lena: I can see how that would be a huge time-saver. If I’m an engineer, I can just run `databricks data-quality run-refresh` to kick off a check after I’ve pushed a new Parquet file to S3. But how does that connect back to the AI analyst? Is the agent itself using the CLI?
18:47 Miles: In a way, yes. Some of these agents, like Claude Code or OpenClaw, are essentially "vibe coding" their way through your infrastructure. They use the Model Context Protocol to "call" these CLI-like tools. For example, if you have a BigQuery MCP server set up, the agent can use an `execute_sql` tool to run a query, interpret the results, and even suggest a fix if the query fails. It’s like having a senior data engineer sitting inside your terminal.
19:13 Lena: That "execute_sql" tool is interesting, but I saw a limitation in the BigQuery MCP docs—it limits query results to 3,000 rows and a three-minute processing time. I guess that’s to keep the "loop" from getting bogged down in massive, unoptimized queries?
0:53 Miles: Exactly. You don't want an AI agent accidentally triggering a million-dollar query that runs for an hour. These limitations are "guardrails" for the loop. They force the agent—and the user—to be more specific. Instead of "Show me all the data," you ask for "the top 10 orders by volume in the last quarter." It keeps the interaction snappy and cost-effective.
19:48 Lena: And if you’re using something like MotherDuck or DuckDB, you can even take that a step further with "local sampling." You use the MCP server to create a small, local DuckDB sample of a massive S3 dataset. The agent iterates on that local sample—which is fast and free—and only once the logic is perfect do you deploy it against the full production data.
20:09 Miles: That is a brilliant "Playbook" move, Lena. It’s about optimizing the feedback loop. We’ve seen that data pipelines are often slow because you’re waiting on large-scale infrastructure. But if the AI is working with a local DuckDB instance through MCP, the feedback is instant—write code, run, see result. It’s like web development for data engineering.
20:27 Lena: I’m also thinking about the "multi-agent" coordination here. If we have 18 specialized agents in our loop, how do they keep from stepping on each other’s toes? Chudi mentioned this in his "bug bounty automation" architecture—where each agent needs a very clear interface for interacting with the system state.
20:45 Miles: That’s the "Context Hygiene" principle. You use "Dev Docs" or specific configuration files to persist the task state so the agents don't get "context amnesia" after a long session. In SvelteKit, you can use `.cursorrules` or specialized YAML files in your Databricks Asset Bundles to define these boundaries. Each agent has a specific "job"—one might be the "Data Profiler," another the "Schema Validator," and another the "Insight Generator."
21:13 Lena: It’s like a well-oiled factory line, but the "product" is intelligence. And because it’s all connected via MCP, they’re all speaking the same language. It reminds me of what Sidelight AI was doing with their "memory system"—where everything is a "Node" and the AI uses MCP to read, search, and update those nodes through conversation.
12:36 Miles: Right. The friction between having a thought and capturing it—or having a data point and analyzing it—is close to zero. The "tagging" layer is where it all compounds. Every metric, every table, every model gets classified. Patterns surface that a human would never spot just by scrolling through a spreadsheet. It’s not "AI magic"—it’s just extremely well-structured data being navigated by a tool that doesn't get tired.