Twitter scraper for agents: search X posts with one token

A founder wants to know what people on X say about a competitor, a launch or a problem. A recruiter wants posts from people who talk about a niche skill. A marketer wants the phrases customers use. All three type "twitter scraper" into a search box, and all three hit the same wall: scripts that break, and a lot of setup before they see one post.

This post shows how an AI agent searches X posts through the looot catalog without any of that on your side. It covers what a search call returns, which providers answer, what a call costs, and how to run a first job safely.

What does a Twitter scraper call return through looot?

A search call takes a keyword or a query and returns a list of matching posts as JSON. Each post has its text, author handle, time, engagement counts where the provider returns them, and the post URL. Some providers also return people results or media results for the same query.

The catalog lists several providers for the main search job, and one entry is X's own API. That entry needs an account you connect yourself, while the others work on the looot token alone. Inspect the endpoint to see which kind it is.

How do I search X without writing a scraper?

You describe the search and your agent runs the endpoint. It finds the endpoint by task, reads the input fields, checks the price and runs it. There is no browser, no login to keep alive and no HTML to parse.

The agent's steps:

1. Search the catalog for "x search posts". Searching is free.
2. Inspect the cheapest endpoint that fits. Read what it returns and what it costs.
3. Run one query with a small limit and read the raw result.
4. Run the full list of queries and save a row per post with the endpoint and cost.

Set up looot with one line first. A new workspace starts at zero balance, so top up before the first run.

What does a Twitter scraper call cost?

You pay per call, and a search call returns a page of posts. The cost of a job is the number of queries times the pages you read. The table reads the cheapest priced endpoint from the live catalog when the page loads.

Use what 1,000 lookups cost to turn that into a job price. Search results vary between providers, so the same query can return different posts. Spot check one query on two providers before you commit to the cheaper one.

Which jobs suit an agent?

Jobs where the agent reads many short posts and sums them up.

• Brand and competitor mentions. Search the name and common misspellings. Ask the agent to group the posts into complaints, questions and praise, and to quote the three most telling ones with their URLs.
• Launch reaction. Search the product name and the hashtag after a launch. Ask for the most repeated objection and the most repeated compliment.
• Customer language. Search a problem phrase, not a product. The words people use in a complaint are the words your landing page should use.
• Lead research. Search a pain point plus a role, and have the agent list the accounts that wrote the best posts. Review the list by hand before you contact anyone.

For a similar job on a forum, the Reddit guide covers comments and subreddit search.

How do I make search results reliable?

Reliability here means two things: the call worked, and the result is a fair sample. The catalog shows a measured success rate for some endpoints, which covers the first. The second is up to you.

Treat each query as a sample, not a census. A search returns a page of posts from the provider's index, and that index does not hold every post. If a count matters, say so in the prompt and ask the agent to run two phrasings of the same query and report the overlap. A low overlap tells you the result is thin.

Choose queries that a person with the problem would actually type. A query built from your own jargon returns posts from people who share your jargon, which is a narrow slice. Write down the time of each call. Posts get deleted and counts move. A row without a call time is hard to defend later.

Keep to public posts. Do not try to collect private accounts, direct messages or content behind a login. Read the platform's terms before you build a product on top of the data, and keep personal data to what the job needs.

What does a finished X research job look like?

Say you sell a scheduling tool and want to know what people hate about their current one. You ask for four queries: "hate my calendar app", "double booked again", "scheduling tool too expensive" and "calendar sync broken". The agent runs one page of each and gets about a hundred posts.

It then drops the posts that are jokes or ads, groups the rest into sync problems, price complaints and missed meetings, and quotes the sharpest post in each group with its URL. You end with three headlines in your customers' own words and a count for each theme, for the price of four calls plus the model's reading time.

Notice what the agent did not do. It did not claim the counts represent all of X. Ask for that caveat in the prompt, and ask the agent to label each count as a count of posts it read.

Run it: a prompt for your agent

What it cannot do

• It reads public posts. It does not post, reply, like or follow.
• It does not return private accounts or direct messages.
• It does not run on a timer. Following a topic over time needs scheduled runs, not available yet, so rerun the search by hand.
• It does not return every post that matches. Results come from a page of the provider's index.
• Fields and sort options differ by provider. Inspect an endpoint before you rely on one.

For the same method on video, see the TikTok guide. To see every X endpoint and its price, open the public catalog.

Last checked 2026-10-03 against the live catalog. Author: Walid Boulanouar.