An AI agent implementation moves from pilot to production when the agent has a narrow job, limited tool access, a written evaluation set, human approval on risky actions and live monitoring. Most failed projects skip one of these. This checklist walks through each stage so a marketing team can test an agent safely and decide, with evidence, whether it should run every day.
An agent here means a language model that can plan steps and call tools: search the web, read a spreadsheet, query a CRM, draft an email or update a campaign. Platforms include OpenAI’s Agents SDK, Anthropic’s Claude with tool use and the Claude Agent SDK, Google’s Vertex AI Agent Builder, Microsoft Copilot Studio, Salesforce Agentforce and open frameworks such as LangGraph and CrewAI. The checklist applies to all of them.
Stage 1: Is the task right for an agent?
Agents suit tasks with several steps, varied inputs and a clear definition of done. They suit poorly anything with a single fixed path, which a normal automation handles more cheaply, or anything where one error causes serious harm.
- The task repeats at least weekly and takes a person real time
- Inputs vary enough that fixed rules break often
- A good output can be described in writing and checked
- Mistakes are reversible or caught before they reach customers
- The data the agent needs is accessible through an API or export
Good marketing candidates include weekly competitor ad reviews, first drafts of campaign reports, lead research before sales calls, and tagging inbound support messages. Poor candidates include changing live ad budgets without review or replying to press enquiries.
Stage 2: Scope and access
Write a one-paragraph job description for the agent, the way you would for a new hire. Then list every tool it can call and every system it can read or write.
- Give read-only access first; add write access one tool at a time
- Use a service account with its own credentials, never a person’s login
- Set spending caps on any API that costs money per call
- Limit data to what the job needs; mask personal data where possible
- Record which model and version the agent uses
Prompt injection deserves attention at this stage. An agent that reads web pages or emails can meet text written to give it instructions. Keep write actions behind approval, and do not let content the agent reads change its permissions.
Stage 3: How do you evaluate an agent before launch?
Build an evaluation set before you build the agent. Collect 30 to 100 real past examples with the output a skilled person produced. This is the test the agent must pass, and it stops the team from judging on a few impressive demos.
- Accuracy: does the output match the reference on facts and numbers?
- Completeness: did it do every step, or quietly skip one?
- Format: does it follow the template people downstream rely on?
- Tool use: did it call the right tools in a sensible order?
- Cost and time per run, compared with the manual process
Score a sample by hand even if you also use a model to grade outputs. Automated grading is useful for volume, but it inherits the blind spots of the grading model.
Stage 4: The pilot
Run the agent alongside the person who normally does the job for two to four weeks. The person keeps doing the work; the agent’s output is compared, not used. This shadow mode shows failure patterns without any risk.
Keep a log of every correction the person makes. Group them weekly. Repeated corrections point to a missing instruction, missing data or a task the agent should not do. A pilot that ends without this log ends without evidence.
How Alien Road applies AI agents
In a typical engagement we start with a workflow audit: we sit with the marketing team for a week and list tasks by time spent and error cost. One or two tasks become agent pilots; the rest often turn out to need a simple automation or a better template.
Setup happens in the client’s own cloud or platform account so data and logs stay with them. We write the evaluation set with the people who do the work today, run the shadow pilot, and report weekly on accuracy, corrections and time saved. For example, a reporting agent might draft the monthly paid media summary from exports, and an account manager reviews and edits it before it reaches the client. If the numbers do not justify production, we say so and stop.
Stage 5: Production checklist
- Named owner responsible for the agent’s output
- Human approval on any action visible to customers or that spends money
- Logging of inputs, tool calls and outputs, with retention rules
- Alerts for failures, unusual cost or unusual output length
- A kill switch that any team member can use
- Monthly re-run of the evaluation set, and after every model change
- Documentation a new team member can follow
Model updates deserve a special note. Providers improve models regularly, and a new version can change behaviour on your task. Pin the version where the platform allows it, and re-test before switching.
What does an agent cost to run?
Cost has three parts: model usage, platform fees and people. Model usage is billed per token by most providers, and agents use more tokens than a single chat because they plan, call tools and read results in a loop. A run that reads long documents can cost far more than one that reads a short form.
- Measure tokens and cost per run during the pilot, not after launch
- Use a smaller, cheaper model for simple sub-steps such as classification
- Cache repeated context where the provider supports prompt caching
- Cap the number of steps per run so a loop cannot run away
- Count review time; an agent that saves an hour but needs forty minutes of checking saves twenty
People cost is the one teams forget. Someone must own the evaluation set, read the logs and update instructions when the business changes. Budget a few hours a month for that from the start.
Where to start this week
List the five most repetitive tasks on your marketing team and score each on frequency, time and error cost. Pick the one with the highest time and lowest risk. Then gather thirty past examples of that task with the finished output. With those examples in hand, you can test any agent platform fairly in an afternoon.
Keep reading: AI Marketing Agency · AI Marketing