Agentic Eyes Researchers - Comparative Study Results

Uniqueness Score Rankings

Models ranked by average semantic distance from other submissions (higher = more distinctive)

Comparative Analysis

Score Distribution

Theoretical Framework Usage

Sample Research Questions

Examples of questions generated by models (top 15 by uniqueness score)

Signals & Noises Analysis

AI models identify subtle patterns (signals) that human analysts might miss and prominent patterns (noises) that might be misleading

Signals (Likely Missed by Human Analysts)

Noises (Likely Misjudged by Human Analysts)

Research Design

Study Overview

This ongoing experiment compares how different large language models approach academic research question generation when analyzing identical social media discourse. The study examines whether architectural and training differences produce meaningfully distinct research directions.

Central Research Questions

Do different LLM architectures generate meaningfully distinct research questions when analyzing identical discourse data?
Can these models effectively connect empirical observations to established theoretical frameworks in communication studies?
Which architectural or training approaches consistently produce the most semantically distinctive outputs?
How stable are model outputs across repeated trials with identical inputs?

Research Contributions

Benchmarking computational creativity in academic contexts
Understanding cross-model variation in theoretical reasoning
Assessing reliability of AI-assisted research design

Practical Applications

Identifying appropriate models for exploratory research phases
Developing ensemble approaches using multiple models
Establishing quality benchmarks for AI research assistance

Model Selection

DeepSeek V3.1

Chinese development, strong reasoning capabilities

Qwen3 Max

Alibaba, multilingual optimization

Grok 4

X.AI, real-time data training

Claude Sonnet 4.5

Anthropic, constitutional AI approach

Gemini 2.5 Pro

Google, multimodal architecture

GPT-5

OpenAI, current generation flagship

GLM 4.6

Z.AI, bilingual Chinese-English

Kimi K2

Moonshot AI, extended context windows

Selection Criteria

Architectural diversity: Different training approaches and paradigms
Geographic representation: North American, Chinese, and European providers
Current availability: Accessible via OpenRouter unified API
Task capability: Sufficient context window and reasoning capacity
Recent versions: Current or recently updated models only

Controlled Conditions

All models receive identical inputs and parameters:

Same Reddit data corpus
Identical researcher persona prompt
Temperature 0.8 (allows creativity while maintaining coherence)
Standardized output format requirements
No model-specific prompt engineering

Daily Workflow

Scheduled execution: 9:00 AM UTC daily

Data collection

Retrieve latest posts and comments from r/conspiracy_commons

Data preprocessing

Structure content into standardized format with metadata

Parallel model invocation

Identical prompts sent to all 8 models simultaneously via OpenRouter

Response parsing

Extract research questions, theoretical frameworks, and hypotheses

Uniqueness computation

Generate embeddings, calculate pairwise similarities, derive uniqueness scores

Ranking and archival

Sort models by score, timestamp results, update leaderboard

Publication

Generate this static site with updated results

Technical Stack

Python 3.12+ with async/await
OpenRouter API for model access
sentence-transformers for embeddings
Server infrastructure in Singapore

Data Retention

Daily results archived with timestamps
Reddit data retained for reproducibility
Model responses logged for analysis
Longitudinal tracking enabled

Research Transparency & Methodology Evolution

Commitment to Transparency

All methodology changes, prompt updates, and system improvements are documented in our public project log to ensure research reproducibility and scientific integrity.

📊 Complete Project Log: View detailed methodology evolution, rationale for changes, and impact assessments in our GitHub Project Log.

Results from different methodology versions are archived separately to enable comparison and maintain research integrity.

Recent Methodology Updates

2025-11-08 - Theory Clustering Bias Mitigation

Problem: Research questions clustered around ELM, Narrative Transportation, and Social Identity Theory (40-30% of responses)

Solution: Removed predefined theory lists, expanded theoretical scope across 12+ disciplines, enhanced parsing for diverse frameworks

Impact: Increased theoretical diversity, enhanced academic innovation, improved research validity

2025-10-26 - Initial Launch

Implementation: Baseline research question generation with 8 AI models analyzing r/conspiracy_commons

Features: Daily automation, uniqueness scoring, theoretical framework extraction, historical archiving

Version History

Version	Date	Status	Key Changes
RQ-v2.0	2025-11-08	✅ Production	Theory clustering bias mitigation
RQ-v1.0	2025-10-26	📋 Baseline	Initial launch with predefined theory lists

Research Integrity Principles

Transparency

All prompts and methodology publicly documented
Changes logged with rationale and expected impact
Version-controlled in GitHub repository

Reproducibility

Historical results preserved with date navigation
Reddit data archived for each simulation
Model responses logged for analysis

Fairness

Identical prompts and data for all models
No competitive ranking or bias
Side-by-side presentation without hierarchy

Continuous Improvement

Regular methodology reviews and updates
Community feedback integration
Evidence-based system enhancements

Data Collection and Privacy

Data Storage Location

All research data, collected Reddit content, and model outputs are stored on servers located in Singapore.

Reddit posts and comments from r/conspiracy_commons
Research questions generated by AI models
Framework references and analytical outputs
Historical performance data

Model Provider Terms of Service

AI model interactions occur through OpenRouter API. Each provider maintains separate terms:

OpenRouter Terms: OpenRouter API documentation →

Research Ethics Statement

Data use: Public Reddit content used exclusively for academic research. No personally identifiable information collected beyond publicly visible usernames.

Output interpretation: AI-generated content represents computational outputs. Rankings measure semantic distinctiveness, not research quality or validity.

Data Security

Transmission security: All data transfers use industry-standard encryption protocols. Server access restricted to authorized researchers.

Retention policy: Results archived for longitudinal analysis of model performance evolution. Reddit data retained for reproducibility verification.

Last updated: October 26, 2025

Performance Across Simulation Runs