# Data Cleaning Scripts

This package includes public data-cleaning scripts under `public-scripts/data-cleaning/`.

They do not call Apidance APIs by themselves and do not need API keys. They clean JSON that you already collected from Apidance Twitter endpoints.

## MCP Resources

MCP clients can read the documentation and script source through resources:

| Resource URI | Description |
| --- | --- |
| `apidance://twitter/data-cleaning` | This data-cleaning guide. |
| `apidance://twitter/scripts/clean-user-tweets.mjs` | Runnable cleaner for token/user timeline datasets. |
| `apidance://twitter/scripts/twitter-normalize.mjs` | Importable Twitter GraphQL normalization helpers. |

## `clean-user-tweets.mjs`

Path:

```text
public-scripts/data-cleaning/clean-user-tweets.mjs
```

Use this script when you have an aggregated per-user timeline file and want compact dashboard-ready JSON.

Typical source endpoints:

| Endpoint id | Apidance path | How the response is used |
| --- | --- | --- |
| `graphql_search_timeline` | `GET /graphql/SearchTimeline` | Finds token/cashtag mentions and candidate users. |
| `graphql_user_tweets_and_replies` | `GET /graphql/UserTweetsAndReplies` | Main source for each candidate user's timeline. |
| `graphql_user_tweets` | `GET /graphql/UserTweets` | Alternative user timeline source. |
| `graphql_user_media` | `GET /graphql/UserMedia` | Optional media-only timeline source. |
| `graphql_tweet_detail` | `GET /graphql/TweetDetail` | Optional reply/quote context source. |
| `graphql_tweet_result_by_rest_id` | `GET /graphql/TweetResultByRestId` | Optional single tweet context source. |
| `custom_api_endpoint` | `GET /graphql/{id}/{path}` | Optional batch tweet context, for example `TweetResultsByRestIds` when available. |

The cleaner expects one JSON file shaped like one of these forms:

```json
{
  "query": {
    "token_symbol": "CASHCAT"
  },
  "users": [
    {
      "user": {},
      "target_token_mentions": {
        "mention_count": 3
      },
      "pages_fetched": 10,
      "tweets": []
    }
  ]
}
```

It also accepts `data.users`, `timelines`, and tweet arrays under `tweets`, `timeline`, `user_tweets`, or `data.tweets`.

Run:

```bash
node public-scripts/data-cleaning/clean-user-tweets.mjs \
  --symbol CASHCAT \
  --input CASHCAT/cashcat_candidate_user_timelines.json \
  --output-dir CASHCAT \
  --public-data-path CASHCAT
```

Optional filters:

```bash
node public-scripts/data-cleaning/clean-user-tweets.mjs \
  --symbol CASHCAT \
  --since 1782835200 \
  --until 1783353600
```

Outputs:

| File | Description |
| --- | --- |
| `<output-dir>/<prefix>_user_tweets_light.json` | Combined compact per-user timeline JSON. |
| `<output-dir>/tweets/<user_id>.json` | One cleaned timeline payload per user. |
| `<output-dir>/tweets/index.json` | Manifest for dashboard-style loading and sorting. |

Cleaned tweet shape:

```json
{
  "id": "tweet id",
  "url": "https://x.com/user/status/id",
  "created_at_iso": "2026-01-01T00:00:00.000Z",
  "created_at_ms": 1767225600000,
  "type": "tweet",
  "text": "tweet text",
  "author": {
    "id": "user id",
    "screen_name": "handle",
    "name": "display name",
    "description": "",
    "followers_count": 0,
    "profile_image_url_https": "",
    "url": ""
  },
  "metrics": {
    "favorite_count": 0,
    "reply_count": 0,
    "retweet_count": 0,
    "quote_count": 0,
    "view_count": 0
  },
  "entities": {
    "symbols": ["TOKEN"],
    "hashtags": [],
    "mentions": [],
    "contract_addresses": []
  },
  "tokens": ["TOKEN"],
  "contracts": ["0x..."],
  "has_token": true,
  "has_contract": true,
  "context": {
    "reply_to": null,
    "quoted": null,
    "retweeted": null
  }
}
```

Per-user output shape:

```json
{
  "schema_version": "1.0",
  "generated_at": "2026-01-01T00:00:00.000Z",
  "query": {
    "token_symbol": "CASHCAT",
    "since_time": 1782835200,
    "until_time": 1783353600
  },
  "user": {},
  "target_token_mentions": {},
  "pages_fetched": 10,
  "stats": {
    "tweet_count": 0,
    "tweets_with_token_count": 0,
    "tweets_with_contract_count": 0,
    "token_count": 0,
    "contract_count": 0
  },
  "token_summary": [],
  "contract_summary": [],
  "tweets": []
}
```

## `twitter-normalize.mjs`

Path:

```text
public-scripts/data-cleaning/twitter-normalize.mjs
```

Use this module when you need reusable normalizers for raw Apidance Twitter GraphQL responses.

Typical source endpoints:

| Endpoint id | Apidance path | Useful exports |
| --- | --- | --- |
| `graphql_user_by_screen_name` | `GET /graphql/UserByScreenName` | `extractUsers`, `normalizeUser` |
| `graphql_user_by_rest_id` | `GET /graphql/UserByRestId` | `extractUsers`, `normalizeUser` |
| `graphql_user_business_profile_team_timeline` | `GET /graphql/UserBusinessProfileTeamTimeline` | `extractUsers`, `extractBottomCursor` |
| `graphql_following` | `GET /graphql/Following` | `extractUsers`, `extractBottomCursor` |
| `graphql_followers` | `GET /graphql/Followers` | `extractUsers`, `extractBottomCursor` |
| `graphql_search_timeline` | `GET /graphql/SearchTimeline` | `extractTweets`, `extractUsers`, `extractBottomCursor` |
| `graphql_user_tweets_and_replies` | `GET /graphql/UserTweetsAndReplies` | `extractTweets`, `extractBottomCursor` |
| `graphql_tweet_detail` | `GET /graphql/TweetDetail` | `extractTweets`, `extractUsers`, `extractBottomCursor` |
| `graphql_tweet_result_by_rest_id` | `GET /graphql/TweetResultByRestId` | `normalizeTweet`, `extractTweets` |
| `graphql_community_tweets_timeline` | `GET /graphql/CommunityTweetsTimeline` | `extractTweets`, `extractUsers`, `extractBottomCursor` |

Example:

```js
import { extractBottomCursor, extractTweets, extractUsers } from "./public-scripts/data-cleaning/twitter-normalize.mjs";

const response = JSON.parse(await fs.readFile("search-page.json", "utf8"));
const users = extractUsers(response);
const tweets = extractTweets(response);
const cursor = extractBottomCursor(response);
```

Exports:

| Export | Output |
| --- | --- |
| `walk(root, visitor)` | Visits every object/array in a JSON-like tree. |
| `extractBottomCursor(root)` | Returns the last bottom timeline cursor. |
| `normalizeUser(userLike)` | Returns a stable normalized user object or `null`. |
| `extractUsers(root)` | Returns de-duplicated normalized users from a full response. |
| `normalizeTweet(tweetLike)` | Returns a stable normalized tweet object or `null`. |
| `extractTweets(root)` | Returns de-duplicated normalized tweets, including embedded reply/quote/retweet context. |

Normalized user shape includes:

```json
{
  "id": "user id",
  "screen_name": "handle",
  "name": "display name",
  "description": "",
  "description_urls": [],
  "location": "",
  "created_at": "",
  "followers_count": 0,
  "friends_count": 0,
  "listed_count": 0,
  "statuses_count": 0,
  "verified": false,
  "verified_type": "",
  "is_blue_verified": false,
  "profile_image_url_https": "",
  "profile_banner_url": "",
  "url": "",
  "professional": null,
  "business_affiliates_count": null,
  "business_affiliation": null
}
```

Normalized tweet shape includes:

```json
{
  "id": "tweet id",
  "url": "https://x.com/handle/status/id",
  "created_at": "",
  "created_at_iso": "",
  "created_at_ms": 0,
  "type": "tweet",
  "text": "",
  "lang": "",
  "conversation_id": "",
  "in_reply_to_status_id": "",
  "quoted_status_id": "",
  "retweeted_status_id": "",
  "author": {},
  "metrics": {},
  "entities": {
    "symbols": [],
    "hashtags": [],
    "mentions": [],
    "urls": [],
    "media": [],
    "contract_addresses": []
  },
  "context": {
    "reply_to": null,
    "quoted": null,
    "retweeted": null
  }
}
```