How We Built a Content Intelligence Loop to Track What Competitors Publish
Competitive intelligence often begins with a deceptively simple question: what did our competitors publish this week?
The first version of the answer sounds easy. Crawl a few websites, collect new URLs, and place them in a dashboard. But a list of links is not competitor analysis. It does not tell a customer whether a rival launched a new product narrative, expanded a recurring content series, earned meaningful third-party coverage, or simply changed a sitemap.
We built Seerly's content intelligence loop to close that gap. It monitors a customer's chosen competitors, finds new material on their websites and across independent publishers, groups related pages into understandable activity, and produces a source-backed weekly read. The aim is not to expose every page we can find. It is to help customers keep abreast of how their market is changing without spending every Monday rebuilding the same research spreadsheet.
This article covers the engineering decisions that made that possible. It does not reproduce every rule or threshold in the implementation. Instead, it focuses on the parts that changed the quality of the product: defining the right unit of analysis, designing discovery as a ladder, treating time as data, separating evidence from interpretation, and making incomplete coverage visible.
1. The hard part of competitor tracking is deciding what counts
Most competitor tracking tools start from URLs. A page appeared, so it becomes an update. That is a reasonable collection model and a poor product model.
Consider three examples:
- A competitor publishes one product announcement that is covered by eight news sites.
- Its editorial team releases twelve pages from a recurring research series.
- A migration places hundreds of old pages into a newly generated sitemap.
Counting URLs turns these into eight events, twelve unrelated posts, and hundreds of new publications. A person looking at the evidence would describe them as one announcement, one editorial push, and probably no meaningful publishing event at all.
That difference shaped the system. The raw page is an evidence record, not always the thing shown to the customer. Our customer-facing unit is a publishing activity: a single page when it stands alone, a recurring series when pages belong together, a third-party story when several publishers cover the same event, or a bulk push when a competitor releases a large related set.
This sounds like a presentation choice. It is actually a data-model choice. Once the product promises to tell customers what competitors are publishing, every later decision depends on what “one thing” means.
It also changes how we think about content intelligence. The system is not trying to predict a universal trend from the whole internet. It is building a defensible view of a defined competitive set, over a defined reporting window, using sources the customer can inspect.
2. We designed the report before we designed the crawler
A useful weekly report needs stable boundaries. If a scan runs on Wednesday, should it compare the previous seven rolling days with the seven before that? What happens to a page published on Sunday but discovered on Tuesday? What happens when a customer adds a new competitor halfway through the week?
We chose complete calendar weeks in the project's timezone. The current partial week waits for the next report. A late discovery can still be included, but it keeps its real publication date rather than pretending it was new when our system happened to find it.
At the start of a cycle, we freeze the monitored set: the customer's brand and the competitors selected for that report. Collection then fans out across that snapshot. This prevents a settings change during processing from producing a report whose totals and evidence disagree.
The cycle has two broad phases:
- Collect evidence for each company. Look at the company's own publishing surfaces and at independent sources elsewhere on the web.
- Build and interpret the report. Normalize pages into activities, calculate coverage and metrics, and then produce a concise analyst read from that fixed evidence.
The distinction matters. Competitor website analysis is an asynchronous, failure-prone workload. One slow domain should not block every other company, and retrying a job should not create a duplicate weekly report. We use an in-flight report as the concurrency boundary, give every collection job an identity tied to that report, and let individual sources degrade without erasing the rest of the week.
The customer sees a coherent observation window even though the underlying work is distributed.
3. Every company has two publishing surfaces
When people ask what a competitor published, they usually mean two different things:
- What the competitor published on its own properties, such as articles, research, announcements, and product education.
- What was published about the competitor elsewhere, such as reporting, interviews, reviews, and other third-party coverage.
We collect these sides separately because their discovery methods and meanings are different. A post on a company's site is direct evidence of what that company chose to say. A news article is evidence of what earned outside attention. Combining them too early produces misleading counts and weakens the story behind a signal.
It also creates a common duplication problem. A company's press page may link to an external article, while web research finds the same article directly. The pipeline resolves that overlap so one piece of coverage does not become two activities.
For customers, the distinction answers two valuable questions at once: what is the competitor trying to establish, and what is the market choosing to amplify?
4. Own-site discovery works as a ladder, not a single crawl
There is no standard endpoint for “show me everything this company published.” Some sites maintain accurate sitemaps with trustworthy modification dates. Some expose RSS or Atom feeds. Some have recognizable content hubs but weak metadata. Others render a useful homepage while blocking deeper automated access.
A crawler that assumes one structure will be accurate on the sites that look like its test fixtures and unreliable everywhere else. We therefore built own-site discovery as a bounded fallback ladder.
Start with declared sources
We first inspect the site's declared publishing infrastructure. Robots files can point to sitemap indexes; sitemaps can reveal dedicated article or newsroom sections; feeds can provide clean entries with dates. These sources are fast and relatively cheap when they are maintained well.
But “present” does not mean “trustworthy.” A sitemap date may reflect a template deployment rather than a substantive page change. A giant sitemap can mix product pages, account routes, documentation, jobs, and articles. The collector classifies likely content paths and evaluates whether the dates are useful before treating them as publishing evidence.
Discover the content roots the site actually uses
Not every content section is called /blog/. It might be /insights/, /learn/, /resources/, /newsroom/, or a branded publication. From the homepage we look for strong publishing labels, clusters of article-shaped links, and declared feeds on the same domain.
We deliberately avoid treating every navigational cluster as content. Documentation, support, status, careers, shops, and application routes have different change patterns and would distort competitor content analysis.
Compare URLs and verify pages when metadata is weak
If dates cannot be trusted, the system compares observed URLs with prior scans and verifies a bounded set of candidate pages. Page-level evidence is slower, so the work has explicit budgets and concurrency limits. A large site cannot monopolize the entire cycle.
This is where production reality forced an important compromise. The most exhaustive scan is not always the most reliable scan. A bounded result with clear coverage is more useful than a job that times out while trying to prove completeness.
Use search as a fallback, not as silent equivalence
When a site refuses access or exposes no useful publishing structure, web search can recover part of the picture. We label that result differently. “Checked through search” is useful evidence, but it is not the same claim as a successful first-party scan.
The ladder lets us monitor competitor blog posts across heterogeneous sites while being honest about how the result was obtained.
5. Third-party discovery needs proof, not just recall
Finding material about a company elsewhere on the web has the opposite failure mode. Recall is easy: a broad query can return many pages. Precision is harder.
A company name can appear in stock listings, directories, syndicated press releases, scraped aggregators, navigation, or an unrelated organization with the same name. Feeding those results directly into an AI summary would create confident noise.
Our third-party path runs multiple research passes, mixes the candidate sets, normalizes URLs, and verifies a bounded sample. It removes known listing and wire patterns, rejects the company's own host, and requires clear evidence that the company is actually the subject of an item.
The goal is not to create the largest possible news feed. It is to build an evidence set that can support statements such as:
- a launch received coverage from independent publishers;
- a comparison page names the customer's brand;
- a topic is gaining attention across multiple sources;
- a competitor appeared on sites that AI answers already cite.
This is a useful distinction between content monitoring and competitive intelligence. Monitoring asks whether a phrase appeared. Intelligence asks what happened, why it matters, and what evidence supports that interpretation.
6. Time boundaries belong in the evidence layer
Publication dates on the web are inconsistent. Pages have published dates, modified dates, feed timestamps, sitemap timestamps, and occasionally no useful date at all. Search can discover an old page today. A publisher can update an article after the event it describes.
If date handling is postponed until the dashboard, the system has already lost the ability to make a defensible weekly claim.
We place activities into report weeks using the strongest available publication evidence. Discovery time is a fallback, not a substitute for a known date. We also carry whether an item was found late, so the interface can explain why something appears now without rewriting history.
Continuity is handled at the project level. Once a page has appeared in a published report, it should not return as a new page in the next report simply because another discovery path found it. Newly added competitors are bounded to the current reporting window so their entire archive does not flood the first report.
These rules do not make dates perfect. They make uncertainty manageable and stop the most common forms of historical inflation.
7. We turn pages into moves customers can understand
After collection, we have evidence records. Before analysis, we convert those records into the units a reader would naturally recognize.
Recurring content becomes a series
Companies often publish through stable templates: research indexes, integration directories, location pages, industry guides, or recurring reports. Showing every page independently makes the feed unreadable and exaggerates the number of strategic moves.
We use URL structure, titles, section identity, timing, and continuity from earlier reports to group related pages. The continuity is important. A recurring series should keep its identity from week to week rather than being renamed by whichever sample happened to appear in the latest scan.
Related coverage becomes one story
Several publishers can cover the same launch. We group third-party articles by company, timing, and headline similarity, then select a representative headline from the strongest available source evidence.
The clustering is intentionally conservative. Joining loosely related stories creates a polished but false narrative. It is better to show two activities than to hide two separate events inside one synthetic theme.
Large releases become a bulk-publishing signal
A sudden group of pages in one section can be a meaningful competitor move. It can also be a migration or a programmatic expansion whose quality we have not measured.
We identify unusually large, tightly timed releases and describe the observable facts: approximate scale, section, timing, and title pattern. We avoid turning those facts into accusations. “Large publishing push” is evidence. “Spam” is an unsupported judgment.
Quality signals are also conditional. We only discuss page length, similarity, or citation presence when enough of the batch was actually measured. Missing measurements should reduce the claim, not invite the model to fill the gap.
This normalization layer is what turns competitor tracking into a readable product. Customers can scan the week as a set of moves rather than processing an audit log of URLs.
8. Deterministic evidence comes before AI interpretation
We use an analyst agent, but we do not ask it to discover the truth from a pile of pages.
The first half of report generation is deterministic. It freezes the weeks and competitor set, applies continuity rules, assigns activities, calculates metrics, records coverage, and creates stable identifiers for every source-backed unit. That report can be published even if the synthesis layer is unavailable.
Only then does the analyst classify relevance, connect activities to tracked topics, identify a small number of signals, and write the weekly read. It is constrained to the supplied evidence and source identifiers. A statement in the brief must point back to the activities that justify it.
That separation gives us three useful properties:
- The evidence survives an AI failure. Customers can still inspect the activity and coverage.
- The narrative is reproducible enough to debug. We can tell whether a problem began in discovery, grouping, or synthesis.
- The model has less room to invent connective tissue. It interprets a bounded report rather than browsing an open-ended corpus.
AI is valuable here because the final product needs prioritization and compression. A customer does not want fifty mechanically generated sentences. They want a short read that covers every active competitor, calls out the few changes worth attention, and remains traceable to sources.
The model improves usability. It does not replace the evidence system.
9. Coverage is part of the product
Competitive intelligence is tempting to present as certainty. A dashboard shows a count, so the count feels complete. In reality, websites refuse requests, feeds go stale, search results vary, and third-party sources disappear.
We represent those conditions explicitly. A source can be checked directly, checked through search, partially checked, stale, or not checked. Customers can see which side of a company was measured and where the system had to fall back.
This has an important product consequence: zero activity and zero visibility are not the same thing.
If a competitor's site was checked successfully and no qualifying page appeared, zero is information. If collection failed, zero would be a misleading claim. The report needs to preserve that distinction all the way to the interface.
The same principle applies to headline metrics. Publisher counts are frozen within a report. Weekly history uses the values appropriate to each observation window instead of naively adding snapshots that may describe overlapping evidence.
Trust is not created by hiding gaps. It is created by showing the limits of the observation and still making the useful portion easy to act on.
10. What the customer sees
The system's internal layers exist to support a much simpler experience.
At the top, customers see the week's competitive activity: unique updates, activity on tracked topics, presence on AI-cited publishers, and updates that name their brand. They can compare competitors, inspect themes over time, and filter the full evidence by source type, topic relevance, and citation context.
The weekly read compresses that evidence into a few source-backed bullets. Signals call attention to events such as a launch, pricing move, comparison, topic push, or meaningful third-party placement. The topic view shows where competitors are increasing coverage and where the customer's own position may need attention.
This is where the content intelligence loop closes. The system is not merely telling customers that competitors published more. It helps them answer:
- Which competitor changed its content strategy?
- Which topics are receiving sustained investment?
- Which stories are earning independent attention?
- Where is our brand being named or compared?
- Which evidence deserves a response, and which can be watched?
The response is not always “publish another article.” A customer might update a product page, strengthen an existing technical guide, create a comparison, brief sales, or decide that a competitor's activity is irrelevant. Good competitive intelligence improves decisions; it does not manufacture a content quota.
11. The tradeoffs we chose deliberately
Building the feature forced several choices that are easy to miss in a polished report.
Bounded collection over theoretical completeness
We cap expensive verification work and allow partial results. This means we may miss a page on an unusually large or hostile site. The alternative is an unbounded cycle that cannot deliver a dependable weekly report.
Conservative grouping over aggressive summarization
We prefer an extra activity to an elegant but incorrect cluster. This keeps the evidence legible and makes later tuning safer.
Source-backed claims over exhaustive prose
The analyst produces a small number of useful signals. It does not turn every detected page into a strategic claim. Scarcity is a feature when the customer's real problem is attention.
Visible uncertainty over false precision
Coverage labels make the interface slightly more complex. They also stop the product from confusing “we observed none” with “we could not observe.” For an engineering system that guides decisions, that honesty is worth the visual cost.
12. What we learned building the loop
The biggest lesson was that competitive content intelligence is not one model or one crawler. It is a chain of modest systems whose boundaries need to agree.
Discovery must preserve source identity. Date handling must happen before weekly aggregation. Deduplication must work across first-party and third-party paths. Grouping must reflect how people describe publishing moves. AI synthesis must be downstream of a report that is useful on its own. Coverage must survive every transformation and reach the customer.
When those pieces line up, competitor analysis becomes more than a periodic research exercise. It becomes a living view of what a market is saying and where attention is moving.
That is the purpose of the feature we built. Customers choose the companies they care about. Seerly keeps watch across owned content and independent coverage, organizes the evidence into a weekly competitive intelligence report, and helps the customer decide what deserves action.
The output is deliberately smaller than the data behind it. That is what makes it useful.

