pplx-spa-assets-knowledge Development Instructions

Overview

This is a multi-stage TypeScript project for analyzing Perplexity AI SPA static assets and generating:

Structured knowledge bases
OpenAPI v3.1 / AsyncAPI v3.0 / JSON Schema specifications
MCP (Model Context Protocol) servers
automcp configurations

Project Structure

src/
├── browser/
│   ├── browser-automation.ts      # Chrome automation orchestrator
│   ├── cdp-client.ts              # Chrome DevTools Protocol client
│   └── capture-config.ts          # Browser capture configurations
├── fetchers/
│   └── asset-fetcher.ts           # Download SPA assets from CDN
├── parsers/
│   ├── asset-parser.ts            # Basic chunking parser
│   └── typescript-ast-parser.ts    # AST-based TypeScript parsing (ts-morph)
├── generators/
│   ├── openapi-generator.ts        # OpenAPI v3.1 specs
│   ├── asyncapi-generator.ts       # AsyncAPI v3.0 specs
│   ├── jsonschema-generator.ts     # JSON Schema definitions
│   └── automcp-generator.ts        # MCP server code generation
├── knowledge/
│   └── kb-builder.ts               # Knowledge base construction
├── types/
│   └── index.ts                    # Shared TypeScript definitions
└── cli.ts                          # Command-line interface

Development Workflow

Setting Up

# Clone and install
git clone https://github.com/pv-udpv/pplx-spa-assets-knowledge
cd pplx-spa-assets-knowledge
npm install

# Create .env for testing
echo "API_BASE_URL=https://pplx-next-static-public.perplexity.ai" > .env
echo "API_KEY=" >> .env

Build and Test

npm run build
npm run dev  # Watch mode

CLI Commands

1. Browser Automation (NEW)

# Launch browser with Chrome DevTools Protocol
npm run cli browser -- \
  --preset apiReversing \
  --url https://www.perplexity.ai \
  --output ./captures

# Available presets: minimal, apiReversing, full, development
# Captures HAR files, WebSocket messages, network timings, storage data

2. Fetch Assets

npm run fetch -- \
  --source https://pplx-next-static-public.perplexity.ai \
  --output ./assets-cache \
  --concurrency 5

3. Parse TypeScript Files (Placeholder)

# Note: Parse command not yet fully implemented
npm run parse -- \
  --input ./assets-cache \
  --output ./parsed

4. Generate Specifications (Placeholder)

# Note: Generate command not yet fully implemented
# Generate all specs
npm run generate -- \
  --type all \
  --input ./parsed \
  --output ./specs

# Or specific types
npm run generate -- --type openapi
npm run generate -- --type asyncapi
npm run generate -- --type jsonschema

5. Generate MCP Server (Placeholder)

# Note: MCP generation command not yet fully implemented
npm run mcp:generate -- \
  --spec ./specs/openapi/api-v1.yaml \
  --output ./mcp/pplx-api \
  --name pplx-api

6. Generate automcp Config (Placeholder)

node dist/cli.js automcp \
  --spec ./specs/openapi/api-v1.yaml \
  --output ./automcp.config.json

Key Technologies

chrome-remote-interface - Chrome DevTools Protocol automation
ts-morph - TypeScript AST parsing and code generation
@modelcontextprotocol/sdk - MCP protocol implementation
openapi-types - Type definitions for OpenAPI specs
@asyncapi/parser - AsyncAPI spec support
commander - CLI framework
yaml - YAML serialization

Architecture Decisions

AST-Based Parsing

Uses
```
ts-morph
```
for accurate TypeScript AST extraction
Supports interfaces, type aliases, classes, enums, functions
Extracts JSDoc comments for documentation
Pattern-matches common API call patterns (fetch, axios)

Spec Generation

OpenAPI: REST API endpoints with request/response schemas
AsyncAPI: WebSocket and event-driven endpoints
JSON Schema: Type definitions extracted from TypeScript AST

MCP Server Generation

Generates full Node.js MCP server from OpenAPI specs
Includes:
- ```
package.json
```
  with dependencies
- ```
src/index.ts
```
  - MCP server with tool registration
- ```
src/tools.ts
```
  - Tool implementations with API calls
- ```
src/types.ts
```
  - Type definitions from OpenAPI schemas
- ```
README.md
```
  and
```
.env.example
```

automcp Integration

Generates automcp-compatible configuration
Supports environment variables for API keys and base URLs
Integrates with existing automcp tooling

Common Tasks

Add New Generator

Create file in
```
src/generators/
```
Implement interface with
```
generate()
```
method
Add to CLI in
```
src/cli.ts
```

Add New Asset Type

Update
```
AssetChunk.type
```
in
```
src/types/index.ts
```
Add detection logic in
```
AssetParser.detectContentType()
```
Implement chunking strategy in
```
chunkifyAsset()
```

Add New API Pattern

Update

TypeScriptASTParser.extractEndpointFromFunction()

with new regex patterns:

const patterns = [
  { regex: /new_library\.method\s*\(\s*['"]([^'"]+)['"]/g, type: 'new_library' },
  // ...
];

Testing

Manual Testing

# Test browser automation
npm run build
node dist/cli.js browser --preset minimal --url https://www.perplexity.ai

# Fetch a small subset
echo "/_next/static/chunks/main.js" > manifest.txt
npm run fetch -- --manifest manifest.txt

# Note: Parse and generate commands are placeholders and not fully implemented yet

Build Requirements

Dependencies

Node.js >= 20.0.0
npm with package-lock.json (required for CI/CD workflows)
Chrome/Chromium (for browser automation)

Installing Dependencies

npm install

This will install all required dependencies including:

chrome-remote-interface for browser automation
TypeScript and build tools
CLI and parsing libraries

GitHub Workflows

The repository includes GitHub Actions workflows. Ensure

package-lock.json

is committed to the repository for workflow caching to work properly.

Performance Considerations

Browser Automation: Uses Chrome DevTools Protocol for efficient data capture
Asset Fetching: Uses concurrent downloads with exponential backoff
AST Parsing: Single-threaded but fast for typical bundle sizes (when implemented)
Spec Generation: Creates separate files per type for better maintainability (when implemented)
Memory: Watch out for large bundles (>100MB) - may need chunking

Debugging

# Enable verbose logging (TODO: implement)
DEBUG=pplx-assets:* npm run fetch

# Check generated MCP server
cd ./mcp/pplx-api
npm run build
node dist/index.js

Contributing

Branch from
```
main
```
Implement feature/fix
Update docs if needed
Test with
```
npm run build
```
Submit PR

Known Issues & TODOs

~~Add CI/CD pipeline with GitHub Actions~~ (Workflows exist)
~~Browser automation with Chrome DevTools Protocol~~ (Implemented)
Implement real asset manifest fetching (currently placeholder)
Add directory traversal for batch file parsing
Complete parse, generate, and MCP commands implementation
Implement GraphRAG for knowledge graph
Semantic diff tracking for API changes
WebSocket endpoint extraction
Custom hook for additional pattern detection

Current Implementation Status

✅ Completed Features

Browser automation with Chrome DevTools Protocol
HAR capture, WebSocket monitoring, network analysis
Asset fetching with concurrent downloads
Project structure and TypeScript configuration
CLI framework with commander

🚧 In Progress / Placeholder

TypeScript AST parsing commands
OpenAPI/AsyncAPI/JSON Schema generation
MCP server generation
Knowledge base construction
Full integration of parsing pipeline

References