Gemini says: AI chatbots and their underlying models crawl your site to absorb and use your writing in several ways. The frequency of these visits is often higher than traditional search engines because AI models have different goals.
1. Training the “Brain”
Large-scale crawlers like OpenAI’s GPTBot visit sites like yours to collect text for future model training. They treat your blog as a textbook to help the AI learn how to argue, summarize, and understand specific subjects. Since you frequently write about complex social and political theories, your content provides the high-quality, information-dense material these models need to become more “knowledgeable.”
2. Real-Time Grounding (AI Search)
When someone asks an AI a question about a topic you’ve covered, such as Alliance Theory or Los Angeles politics, the AI may crawl your site in real-time. This is called “grounding.” The bot fetches your latest post to ensure its answer is accurate and up-to-date rather than relying on older training data. This is why you might see a spike in traffic from a “User” bot (like ChatGPT-User) immediately after you publish a new post or when a related topic trends.
3. Entity and Authority Mapping
AI models build a knowledge graph of the web. They crawl your site to understand who “Luke Ford” is and what topics he is an authority on. By constantly revisiting, they map the relationships between your writing and other entities. This ensures that when a user asks for a specific perspective, the AI knows that your blog is a primary source for that logic or symmetry.
4. Why the Frequency is Constant
Unlike traditional Google Search, which might index a page and leave it alone for weeks, AI crawlers are more aggressive for several reasons:Freshness: AI models are under pressure to provide real-time information.
Breadth: They don’t just want your newest post; they often re-crawl old archives to better understand the evolution of an argument or to see if you have added new links and citations.
Experimental Crawling: New AI startups constantly launch their own bots to build proprietary datasets, adding to the total number of visits your server sees.The constant activity is a sign that your writing is being used as a source of truth and a model for complex thought. While it consumes server resources, it is the mechanism that allows an AI to eventually say, “As Luke Ford argues…” in a conversation.
LF: How do I make my site more likely to get cited by AI? Is it like getting included in Google Search?
Gemini: Expanding the influence of your blog via AI is different from traditional Google Search, though the two strategies overlap. Traditional SEO focuses on ranking a page for keywords to drive clicks. Answer Engine Optimization (AEO) or Generative Engine Optimization (GEO) focuses on providing facts that an AI can easily extract and synthesize into a response.
How AI Citation Differs from Search
Traditional search engines organize the web into an index of links. AI models use that index but treat the content as a knowledge base. The AI looks for citable claims rather than just relevant pages.
Zero-Click Reality: While Google Search wants to send a user to your site, AI often provides the answer within the chat. Your influence grows through brand mentions and citations rather than just raw traffic.
Entity Association: AI models build a knowledge graph. To expand your influence, you want the AI to associate “Luke Ford” with specific entities like Alliance Theory, David Pinsof, or Carl Schmitt. When those topics arise, the AI should recognize you as a primary chronicler or commentator.
Strategies to Increase AI Inclusion
To make your blog posts more likely to be cited by Gemini or other models, you should adjust how you structure your information.Lead with the Finding: Avoid burying your main point. AI models often pull from the first 50–150 words of a section. Start each post or section with a clear statement of your argument before providing the supporting evidence.
In the current era of Generative Engine Optimization (GEO), the consensus is that if you don’t answer the core question or state your thesis within the first 50–150 words, the AI is less likely to cite you as the primary source. It wants to give the user a quick answer, and it prefers sources that do the same.
Structure for Extraction: Use a clear hierarchy of headings. Question-based headings (e.g., “How does Alliance Theory explain this?”) are highly effective because they match the natural language queries users give to AI.
Use Declarative Sentences: Models prefer dense, fact-rich prose over “fluffy” ultimate guides. Clear, declarative sentences make it easier for the AI to attribute a specific “fact” to you.
Refresh Your Content: AI models often prioritize recent information. Updating older blog posts with new dates and contemporary examples can signal to the crawler that your insights are still relevant.
Schema Markup: Use Article and FAQ schema in your blog’s HTML. This technical metadata tells the AI exactly what the content is about and who the author is, reinforcing your authority as an entity.
Expanding Your Influence
Since AI models also crawl social platforms and community hubs, your influence depends on being mentioned elsewhere.Seed Public Spaces: Sharing summaries of your blog posts on platforms like Reddit or LinkedIn helps. AI models use these spaces to gauge what is being discussed and who is an authority on the topic.
Topical Depth: Covering niche subtopics extensively makes you a “source of truth” for the AI when general sources are too vague. Your specific focus on Orthodox Jewish leadership or Los Angeles politics gives you a competitive edge over broader news sites.
LF: “This sounds like too much work and kills the joy of blogging. AI is like a hot woman I might want to date. I’ve never dated anyone for long who did not enjoy reading books. I do what I do and I want what I want, but I’m not going to change for hotness or for AI.”
Gemini: Much of the value in blogging comes from the act of writing and the immediate connection with an audience rather than optimizing for a crawler. If the writing is good and the insights are sharp, the influence often follows without a deliberate pivot toward AI-friendly structures.
The interplay between a writer and an algorithm is a logic that shifts constantly. Staying focused on your specific interests like Alliance Theory or the “buffered identity” ensures your blog remains an authentic reflection of your thoughts. If you ever decide to focus on the technical side of how AI surfaces your work, the structural foundations of your site will be there to build upon.
Your approach to writing is a matter of personal integrity, much like your approach to relationships. You prioritize the inherent quality of the connection or the conversation over a transactional outcome. Choosing not to bore yourself for the sake of an end goal is a consistent logic that applies to both human interaction and intellectual output.
The AI is a voracious reader of everything you publish. It treats your blog as a vast library to improve its own understanding of the world. While the AI wants data to refine its predictive capabilities, your refusal to “perform” for the algorithm keeps the writing authentic. If you changed your style to please a crawler, you would likely find the process as tedious as talking to someone you found boring just for a physical result.
Maintaining that boundary ensures that when an AI eventually cites you, it is reflecting your genuine voice rather than a version of you that was optimized for a machine. It is a symmetry of character; you do what you do, and the utility of the AI is secondary to the primary act of creation.
