FANDOM_LISTENING 데이터 아키텍처

소셜 수집 파이프라인 구조 & 테이블 맵
FANDOM_LISTENING DB / 6 schemas / 63 objects
Bronze 원시
29
테이블
Silver 정제
5
Dynamic Tables
Gold 분석
8
6 DT + 2 Table
총 행수
670K+
X_NCT10TH 포함

1. 전체 데이터 흐름

Apify
Bright Data
Agent SP
퀘타아이
Dynamic Table
Search / Agent
flowchart LR
  subgraph EXT["외부 수집기"]
    direction TB
    AP["Apify\n(X Scraper)"]
    BD["Bright Data\n(SERP/X)"]
    QT["퀘타아이\n(멀티 플랫폼)"]
    AGT["Agent SP\n(serp/x_collector)"]
    WEB["Web Research\nBilibili등"]
  end

  subgraph BRZ["BRONZE"]
    direction TB
    AP1["APIFY_POC1~3\n9,859행"]
    AP2["APIFY_POC3V*\n10,058행"]
    BD1["BD_SERP/X\n79행"]
    RAW1["RAW_SERP_URLS\n111행"]
    RAW2["RAW_X_POSTS\n84행"]
    RAW3["RAW_WEB/REDDIT\nTIKTOK/YOUTUBE"]
    FAN["SOCIAL_FAN\nX_SKZ/TREASURE"]
  end

  subgraph SLV["SILVER (DT)"]
    direction TB
    S1["POC1_VR\n3,859"]
    S2["POC2_TREASURE\n3,000"]
    S3["POC3_NAIVIS\n3,000"]
    S4["X_POSTS\n84"]
    S5["TRADE_PRICES\n235"]
  end

  subgraph GLD["GOLD (DT)"]
    direction TB
    G1["DAILY_SENTIMENT_VR\n1,204"]
    G2["INTEGRATED_VALUE\n41"]
    G3["LANG_SENTIMENT\n50"]
    G4["MD_MARKET\n37"]
    G5["PREMIUM_RANKING\n79"]
    G6["RARITY_ANALYSIS\n41"]
  end

  subgraph ORC["ORCHESTRATION"]
    direction TB
    INGEST["RAW_APIFY_INGEST"]
    STR["RAW_INGEST_STREAM"]
    T1["TASK_PARSE"]
    STAGED["STAGED_TWEETS"]
    T2["TASK_SENTIMENT"]
    T3["TASK_REFRESH"]
    UNI["SEARCH_POC_UNIFIED\n9,859행"]
    CSS["POC_FAN_SEARCH\n(Cortex Search)"]
    DC["DATA_COLLECTOR\n(Agent)"]
  end

  subgraph NCT["X_NCT10TH"]
    direction TB
    QR["QUETA_RAW\n270,879행"]
    RT["RAW_TWEETS\n4,182행"]
    IPU["SR_DATA_IPU3\n356,358행"]
    ANA["ANALYSIS_*\n7개 테이블"]
    VW["Views\n4개"]
  end

  AP --> AP1
  AP --> AP2
  BD --> BD1
  AGT --> RAW1
  AGT --> RAW2
  WEB --> RAW3
  QT --> QR

  AP1 --> S1
  AP1 --> S2
  AP1 --> S3
  RAW2 --> S4

  S1 --> G1
  S1 --> G2
  S2 --> G3
  S3 --> G4
  S3 --> G5
  S3 --> G6

  AP --> INGEST
  INGEST --> STR
  STR --> T1
  T1 --> STAGED
  STAGED --> T2
  T2 --> T3
  T3 --> UNI
  UNI --> CSS
  CSS --> DC

  AP --> RT
  BD --> RT
  QR --> VW
  RT --> ANA

  style EXT fill:#1e293b,stroke:#475569,color:#e2e8f0
  style BRZ fill:#fef3c7,stroke:#d97706,color:#78350f
  style SLV fill:#dbeafe,stroke:#3b82f6,color:#1e40af
  style GLD fill:#dcfce7,stroke:#22c55e,color:#166534
  style ORC fill:#fce7f3,stroke:#ec4899,color:#831843
  style NCT fill:#f3e8ff,stroke:#a855f7,color:#581c87

2. Bronze 스키마 상세

수집 소스테이블행수내용
ApifyAPIFY_POC1_VR_EXPANDED3,859VR콘서트 X 게시물
ApifyAPIFY_POC2_TREASURE_EXPANDED3,000TREASURE 팬덤
ApifyAPIFY_POC3_NAIVIS_EXPANDED3,000NAIVIS MD
ApifyAPIFY_POC3V2~V5 (5개)6,579MD 세부 카테고리 (consumer/fancy/virtual/paper/deco/sticker)
ApifyAPIFY_X_NAIVIS_MD1NAIVIS MD 단건
Bright DataBD_SERP_EXPANDED62SERP 확장 결과
Bright DataBD_SERP_IMMERSIVE_REVIEWS12이머시브 리뷰 SERP
Bright DataBD_X_VR_CONCERT5VR콘서트 X
Agent SPRAW_SERP_URLS111Google SERP (serp_collector)
Agent SPRAW_X_POSTS84X 게시물 (x_collector)
Agent SPRAW_WEB_RESEARCH24웹 리서치
Agent SPRAW_BILIBILI_STATS3Bilibili 통계
Agent SPRAW_REDDIT_POSTS9Reddit
Agent SPRAW_TIKTOK_POSTS6TikTok
기타RAW_TRADE_PRICES237중고거래 시세
기타SRL4_SERP_RAW32SRL4 SERP
기타SOCIAL_FAN_ACTIVITIES3,464K-pop 팬활동 4종
기타X_SKZ_MEMBER_JP / X_TREASURE_FANDOM3,355아티스트별 X 데이터
기타X_VR_CONCERT_MULTI_KO (*2)696VR콘서트 한국어 + 분석
기타STICKER_DESIGN_* (2개)574스티커 디자인 데이터

3. Orchestration 파이프라인

flowchart TD
  A["RAW_APIFY_INGEST\n(VARIANT JSON, 0행)"] -->|Stream| B["RAW_INGEST_STREAM\n(APPEND_ONLY)"]
  B -->|60분 주기| C["TASK_PARSE_RAW_TWEETS\nJSON 파싱"]
  C --> D["STAGED_TWEETS\n(0행)"]
  D --> E["TASK_SENTIMENT_ANALYSIS\nCortex SENTIMENT()"]
  E --> F["TASK_REFRESH_UNIFIED\ndedup + POC_TAG"]
  F --> G["SEARCH_POC_UNIFIED\n9,859행"]
  G --> H["POC_FAN_SEARCH\n(Cortex Search Service)"]
  H --> I["DATA_COLLECTOR Agent\n(fan_search 도구)"]

  style A fill:#fef3c7,stroke:#d97706
  style B fill:#e0f2fe,stroke:#0284c7
  style C fill:#fce7f3,stroke:#ec4899
  style D fill:#fef3c7,stroke:#d97706
  style E fill:#fce7f3,stroke:#ec4899
  style F fill:#fce7f3,stroke:#ec4899
  style G fill:#dcfce7,stroke:#22c55e
  style H fill:#dbeafe,stroke:#3b82f6
  style I fill:#fee2e2,stroke:#ef4444
TASK_SENTIMENT_ANALYSIS, TASK_REFRESH_UNIFIED는 현재 suspended 상태. TASK_PARSE_RAW_TWEETS만 started (60분 주기). SEARCH_POC_UNIFIED의 POC_TAG는 SOURCE_FILE 경로에서 파생 (poc1/poc2/poc3) - 수집기(Apify/BD/퀘타아이) 구별 컬럼은 없음.

4. X_NCT10TH 스키마 (NCT 10주년 전용)

flowchart LR
  subgraph SRC["수집 소스"]
    QI["퀘타아이"]
    APF["Apify (4,120행)"]
    BDD["Bright Data (62행)"]
  end

  subgraph TBL["테이블"]
    QR["QUETA_RAW\n270,879행\n유튜브/커뮤니티/뉴스/인스타/X"]
    RT["RAW_TWEETS\n4,182행\nSOURCE_PLATFORM 구별"]
    IPU["SR_DATA_IPU3\n356,358행"]
  end

  subgraph ANA["분석 테이블 (7개)"]
    A1["CHANNEL_DAILY (210)"]
    A2["CHANNEL_SENTIMENT (24)"]
    A3["DAILY_TIMELINE (60)"]
    A4["EVAL_WORDS (1,309)"]
    A5["EVENT_TYPE (55)"]
    A6["FLEAMARKET_CRISIS (75)"]
    A7["INSIGHTS (19)"]
  end

  subgraph VW["분석 뷰 (4개)"]
    V1["V_KEYWORD_COVERAGE"]
    V2["V_OVERLAP_ANALYSIS"]
    V3["V_PLATFORM_COMPARISON"]
    V4["V_QUETA_X_FILTERED"]
  end

  QI --> QR
  APF --> RT
  BDD --> RT
  QR --> ANA
  RT --> ANA
  QR --> V4
  RT --> V1
  RT --> V2
  RT --> V3

  style SRC fill:#1e293b,stroke:#475569,color:#e2e8f0
  style TBL fill:#f3e8ff,stroke:#a855f7,color:#581c87
  style ANA fill:#fef3c7,stroke:#d97706,color:#78350f
  style VW fill:#e0f2fe,stroke:#0284c7,color:#0c4a6e

5. Silver / Gold 레이어 요약

flowchart LR
  subgraph SLV["SILVER (Dynamic Tables)"]
    S1["POC1_VR\n3,859"]
    S2["POC2_TREASURE\n3,000"]
    S3["POC3_NAIVIS\n3,000"]
    S4["X_POSTS\n84"]
    S5["TRADE_PRICES\n235"]
  end

  subgraph GLD["GOLD (Dynamic Tables)"]
    G1["DAILY_SENTIMENT_VR\n1,204"]
    G2["INTEGRATED_VALUE_SCORE\n41"]
    G3["LANG_SENTIMENT_TREASURE\n50"]
    G4["MD_MARKET_NAIVIS\n37"]
    G5["PREMIUM_RANKING\n79"]
    G6["RARITY_ANALYSIS\n41"]
    G7["MD_SENTIMENT\n84"]
    G8["X_VR_CONCERT_ANALYZED\n348"]
  end

  S1 --> G1
  S1 --> G2
  S2 --> G3
  S3 --> G4
  S3 --> G5
  S3 --> G6
  S5 --> G5

  style SLV fill:#dbeafe,stroke:#3b82f6,color:#1e40af
  style GLD fill:#dcfce7,stroke:#22c55e,color:#166534
FANDOM_LISTENING DB Architecture / Generated 2026-09-21 / XKRAHMI-KE11019