When initially experimenting with LLMs and agentic AI, software program engineers at Notion AI utilized superior code era, complicated schemas, and heavy instructioning.
Shortly, although, trial and error taught the crew that it might eliminate all of that difficult information modeling. Notion’s AI engineering lead Ryan Nystrom and his crew pivoted to easy prompts, human-readable representations, minimal abstraction, and acquainted markdown codecs. The end result was dramatically improved mannequin efficiency.
Making use of this re-wired method, the AI-native firm launched V3 of its productiveness software program in September. Its notable function: Cutomizable AI brokers — which have shortly turn out to be Notion’s most profitable AI software thus far. Based mostly on utilization patterns in comparison with earlier variations, Nystrom calls it a “step perform enchancment.”
“It's that feeling of when the product is being pulled out of you moderately than you attempting to push,” Nystrom explains in a VB Past the Pilot podcast. “We knew from that second, actually early on, that we had one thing. Now it's, ‘How might I ever use Notion with out this function?’”
‘Rewiring’ for the AI agent period
As a conventional software program engineer, Nystrom was used to “extraordinarily deterministic” experiences. However a light-weight bulb second got here when a colleague suggested him to easily describe his AI immediate as he would to a human, moderately than codify guidelines of how brokers ought to behave in numerous situations. The rationale: LLMs are designed to know, “see” and purpose about content material the identical means people can.
“Now, every time I'm working with AI, I’ll reread the prompts and gear descriptions and [ask myself] is that this one thing I might give to an individual with no context they usually might perceive what's happening?” Nystrom stated on the podcast. “If not, it's going to do a foul job.”
Stepping again from “fairly difficult rendering” of knowledge inside Notion (similar to JSON or XML) Nystrom and his crew represented Notion pages as markdown, the favored device-agnostic markup language that defines construction and that means utilizing plain textual content with out the necessity for HTML tags or formal editors. This enables the mannequin to work together with, learn, search and make modifications to textual content recordsdata.
In the end, this required Notion to rewire its methods, with Nystrom’s crew focusing largely on the middleware transition layer.
Additionally they recognized early on the significance of exercising restraint in relation to context. It’s tempting to load as a lot info right into a mannequin as attainable, however that may sluggish issues down and confuse the mannequin. For Notion, Nystrom described a 100,000 to 150,000 token restrict because the “candy spot.”
“There are circumstances the place you possibly can load tons and tons of content material into your context window and the mannequin will battle,” he stated. “The extra you place into the context window, you do see a degradation in efficiency, latency, and in addition accuracy.”
A spartan method can also be essential within the case of tooling; this will help groups keep away from the “slippery slope” of countless options, Nystrom suggested. Notion focuses on a “curated menu” of instruments moderately than a voluminous Cheesecake Manufacturing facility-like menu that creates a paradox of alternative for customers.
“When individuals ask for brand spanking new options, we might simply add a software to the mannequin or the agent,” he stated. However, “the extra instruments we add, the extra choices the mannequin has to make.”
The underside line: Channel the mannequin. Use APIs the way in which they had been meant for use. Don't attempt to be fancy, don't attempt to overcomplicate it. Use plain English.
Take heed to the complete podcast to listen to about:
-
Why AI continues to be within the pre-Blackberry, pre-iPhone period;
-
The significance of "dogfooding" in product growth;
-
Why you shouldn’t fear about how price efficient your AI function is within the early levels — that may be optimized later;
-
How engineering groups can hold instruments minimal within the age of MCP;
-
Notion’s evolution from wikis to full-blown AI assistants.
Subscribe to Past the Pilot on Apple Podcasts, Spotify, and YouTube.
Keep forward of the curve with NextBusiness 24. Discover extra tales, subscribe to our publication, and be a part of our rising group at nextbusiness24.com