dewemoji/rebuild-progress.md

# NativePHP Rebuild Progress (with `dewemoji-live` audit)

## Confirmed source folders

- Backend: `../dewemoji-api`
- Live backend reference: `../dewemoji-live-backend` (source of truth for current API + community/pro flows)
- Chrome extension: `../dewemoji-chrome-ext`
- Website (legacy scaffold): `../dewemoji-site`
- Live production reference: `../dewemoji-live` (source of truth for parity)

## Current baseline

- New rebuild app: `app/` (Laravel + NativePHP Desktop).
- API v1 routes exist in rebuild: `/v1/emojis`, `/v1/categories`, `/v1/license/verify`.
- Website routes currently in rebuild: `/`, `/emoji/{slug}`, `/api-docs`, `/pricing`, `/privacy`, `/terms`.
- Live site has additional behavior and SEO/route details that must be ported before full parity.

## Agreed strategy (locked)

- Build order: backend-first, then frontend integration.
- Community feature: included in migration scope, but implemented last after core/pro stabilization.
- Payment/provider mode: start in sandbox, document switch path to live (`SANDBOX -> LIVE`) in project docs.
- Database: fresh Laravel-first schema + import scripts from legacy data sources.
- Metrics endpoints: keep, but internal-only (admin token/IP allowlist), not public.
- Upgrade policy: migration is parity-first, but we will take safe opportunities to improve architecture, security, and observability.

## Phase checklist

### Phase 0 - Retrace and documentation
- [x] Revalidated source folders with corrected names.
- [x] Rebuilt initial docs against corrected folders.
- [x] Added live audit from `dewemoji-live`.
- [x] Added backend audit from `dewemoji-live-backend`.

### Phase 1 - Foundation
- [x] Initialize NativePHP app in `dewemoji/app`.
- [x] Install NativePHP Desktop scaffolding.
- [x] Define env/config strategy.
- [x] Use canonical emoji dataset as baseline.

### Phase 2 - API parity (extension first)
- [x] Implement `GET /v1/emojis`.
- [x] Implement `GET /v1/categories`.
- [x] Implement `POST /v1/license/verify` (temporary env-based validation).
- [x] Support both `q` and `query`.
- [ ] Add/verify full response contract parity with live docs (`variants`, `related`, trimmed `keywords_en`, limit behavior by tier).
- [x] Match live cache/rate semantics baseline (`page=1` metering behavior, 401/429 payload shape, ETag/304 behavior).
- [x] Verify header compatibility baseline: `Authorization`, `X-License-Key`, `X-Account-Id`, `X-Dewemoji-Frontend`, `X-Dewemoji-Tier`.
- [x] Restrict CORS to configured origins (no default `*`).
- [x] Add missing live backend routes/contracts now present in production API:
  - `/v1/license/activate`
  - `/v1/license/deactivate`
  - `/v1/health`
  - `/v1/metrics` and `/v1/metrics-lite` (internal/admin decision needed: keep, secure, or remove)
- [x] Reconcile live route mismatch: added `/v1/emoji` and `/v1/emoji/{slug}` in rebuild API.
- [x] Add sandbox/live provider switch documentation and env examples (`BILLING_MODE=sandbox|live`, keys, callbacks, smoke test flow).

### Phase 3 - Website parity from `dewemoji-live`
- [x] Core pages exist in rebuild.
- [ ] Add missing pages/routes: `/support`, `/browse`, pretty category routes (`/{category}` and `/{category}/{subcategory}`).
  - [x] `/browse`, `/{category}`, `/{category}/{subcategory}` implemented in rebuild.
- [ ] Keep URL behavior parity (canonical no-trailing-slash pages, redirect rules, pretty-to-query hydration).
  - [x] no-trailing-slash redirect middleware and canonical link baseline implemented.
  - [x] pretty route hydration wired into homepage initial filters + URL sync.
- [ ] Port homepage behavior parity:
  - API-backed filters (`q`, category, subcategory), URL sync, load-more pagination.
  - API fallback when scoped search returns 0 (retry on `all` + hint).
- [ ] Port single emoji page parity:
  - 404 `noindex` for missing.
  - 410 + `X-Robots-Tag: noindex, noarchive` for policy-hidden emoji.
  - skin-tone variant logic and optional tone path (`/emoji/{slug}/{tone}`).
  - related fallback (same subcategory), prev/next navigation.
  - details blocks: aliases, shortcodes, EN/ID keywords, copy interactions.
  - curated blurbs support from `data/emoji_descriptions.json`.
- [ ] Port legal/support content parity and FAQ schema blocks.

### Phase 4 - Pricing and payments
- [ ] Keep pricing structure parity:
  - Free, Pro subscription, Lifetime.
  - Pro: `$3/mo` and yearly display `$27/yr` in UI.
  - Lifetime: `$69`.
- [ ] Preserve live Gumroad links:
  - `https://dwindown.gumroad.com/l/dewemoji-pro-subscription`
  - `https://dwindown.gumroad.com/l/dewemoji-pro-lifetime`
- [ ] Keep IDR/Mayar messaging parity (manual-renew note).
- [ ] Implement real license lifecycle (activate/deactivate/verify + max 3 Chrome profiles) in new backend.
- [x] Implement real license lifecycle baseline in rebuild (`verify/activate/deactivate`, immutable owner binding behavior, max device cap).
- [ ] Implement provider verification parity:
  - [x] Baseline service layer + env/config wiring + safe HTTP fallback
  - [x] `/v1/license/verify` contract hardening: provider details + diagnostics (`details.gumroad`, `details.mayar`)
  - [ ] Gumroad verify API flow (final payload/contract parity with live provider account)
  - [ ] Mayar verify API flow (final payload/contract parity with live provider account)
  - gateway mode switch (`sandbox` vs `live`)
- [ ] Implement immutable license binding to user + multi-device activation policy parity.

### Phase 5 - SEO parity (must not disrupt GSC)
- [ ] Preserve canonical strategy for all pages (including emoji detail + pretty category pages).
- [ ] Add/verify meta + social tags parity: title/description/OG/Twitter + theme color.
- [ ] Port JSON-LD strategy:
  - Global `WebSite` + `SearchAction` + Organization.
  - `TechArticle` on `/api-docs`.
  - `Product` + `FAQPage` on `/pricing`.
  - `FAQPage` on `/support`.
  - `CreativeWork` + `BreadcrumbList` on emoji pages.
- [ ] Implement `robots.txt` parity and dynamic `sitemap.xml` parity.
- [ ] Ensure sitemap excludes policy-hidden emoji URLs (same filter policy as live).
- [ ] Keep core indexed URLs stable: `/`, `/pricing`, `/api-docs`, `/support`, `/privacy`, `/terms`, `/emoji/{slug}`.

### Phase 6 - Analytics, consent, and compliance
- [ ] Re-implement cookie consent flow before analytics activation.
- [ ] Re-implement GA4 only on allowed production hosts (live uses `G-R7FYYRBVJK`).
- [ ] Keep privacy/terms statements aligned with live content.

### Phase 7 - Data/ops pipelines
- [ ] Port blurb pipeline:
  - `jobs/seed_blurbs_from_dataset.php`
  - `jobs/sync_blurbs.php` (NocoDB approved blurbs sync).
- [ ] Define NativePHP/Laravel replacement for live file microcache (`cache/emoji/*.html`) if still needed for SEO performance.
- [ ] Add rebuild-side commands/jobs for sitemap regeneration and cache warmup.
- [ ] Port backend dataset pipelines from `dewemoji-live-backend/jobs`:
  - JSON -> SQL import (`import_emojis_json_to_sql.php`)
  - SQL -> JSON build (`build_emojis_json_from_sql.php`)
  - Keywords index build (`build_keywords_json_from_sql.php`)
  - Unicode parity validation (`validate_emojis_against_unicode.php`)
  - License expiry revocation cron (`check_license_expiry.php`)

### Phase 8 - Community feature migration
- [ ] Port contributor auth flow:
  - magic link token issue/verify (`/v1/contrib/auth/request`, `/v1/contrib/auth/verify`)
  - stateless HMAC token strategy or Laravel equivalent.
- [ ] Port contribution flows:
  - suggest keywords (`/v1/contrib/suggest`)
  - private -> public promotion (`/v1/contrib/make-public`)
  - list and search (`/v1/contrib/list`, `/v1/contrib/search`)
  - voting + pending queue (`/v1/contrib/vote`, `/v1/keywords/pending`)
- [ ] Port moderation protections:
  - Turnstile verification
  - AI guard moderation pipeline (OpenRouter + usage caps/cache)
  - Redis/APCu rate limiting (vote/suggest/publish paths)
- [ ] Port auto-moderation behavior:
  - score thresholds (`vote_auto_approve_score`, `vote_auto_reject_score`)
  - status/visibility transitions (`private`, `public_pending`, `public`, `approved`, `rejected`)
- [ ] Fix live bug during migration: `/v1/contrib/search` is nested under `/v1/keywords/pending` block in current controller, so route behavior should be revalidated in rebuild.

### Phase 9 - Extension integration and release
- [ ] Point `dewemoji-chrome-ext` to new API host.
- [ ] Validate free/pro flow end-to-end with real license checks.
- [ ] Run parity QA on tone handling, insert/copy behavior, and API limits.
- [ ] Prepare migration + rollback checklist (DNS/host switch, redirects, monitoring).

## Recent implementation update

- Added new API endpoints in rebuild:
  - `GET /v1/emoji`
  - `GET /v1/emoji/{slug}`
  - `POST /v1/license/activate`
  - `POST /v1/license/deactivate`
  - `GET /v1/health`
  - `GET /v1/metrics`
  - `GET /v1/metrics-lite`
- Added internal-protected metrics controller (`token` or IP allowlist).
- Added sandbox/live billing mode documentation: `billing-sandbox-live.md`.
- Added fresh Laravel migrations for core backend state:
  - `licenses`
  - `license_activations`
  - `usage_logs`
- Added `LicenseVerificationService` and wired controllers to use one verification path:
  - sandbox mode
  - live key-list mode
  - baseline Gumroad/Mayar provider calls (+ local stub test keys)
- Added SEO-safe route/canonical baseline:
  - `/browse` route
  - pretty category routes (`/{category}`, `/{category}/{subcategory}`)
  - trailing slash -> canonical path redirect (301)
  - canonical `<link>` output from layout

## Live audit highlights (reference)

- Live web routes in `dewemoji-live/public_html`: `/`, `/emoji/{slug}`, `/browse`, `/pricing`, `/api-docs`, `/support`, `/privacy`, `/terms`, `/sitemap.xml`.
- Rewrite rules and canonicalization live in `dewemoji-live/public_html/.htaccess`.
- SEO assets:
  - `dewemoji-live/public_html/includes/head.php`
  - `dewemoji-live/public_html/sitemap.xml.php`
  - `dewemoji-live/public_html/robots.txt`
- Emoji page implementation reference:
  - `dewemoji-live/public_html/emoji.php`
  - includes microcache + structured data + policy filtering.
- Pricing + payment references:
  - `dewemoji-live/public_html/pricing.php`
  - `dewemoji-live/public_html/support.php`
  - `dewemoji-live/public_html/privacy.php`
  - `dewemoji-live/public_html/terms.php`
- Blurb data + jobs:
  - `dewemoji-live/public_html/data/emoji_descriptions.json`
  - `dewemoji-live/jobs/sync_blurbs.php`
  - `dewemoji-live/jobs/seed_blurbs_from_dataset.php`

## Live backend audit highlights (`dewemoji-live-backend`)

- Backend architecture is a custom PHP router (`public_html/public/index.php`) with controller-per-endpoint files and shared helpers.
- Main live API surface discovered:
  - `/v1/emojis`, `/v1/categories`
  - `/v1/license/verify`, `/v1/license/activate`, `/v1/license/deactivate`
  - `/v1/contrib/*` community endpoints (suggest/list/vote/auth/make-public/search)
  - `/v1/keywords/pending`
  - `/v1/health`, `/v1/metrics`, `/v1/metrics-lite`
- Data mode is hybrid:
  - API reads from JSON dataset (`public_html/app/data/emojis.json`) for emoji search.
  - Licensing, usage logs, and community data read/write in MySQL.
  - Redis/APCu/in-memory used for runtime rate-limiting fallback chain.
- Pro logic currently exists in live backend:
  - plan resolution from license key + first-party whitelist headers/origin.
  - pro/free limits for API and contribution quotas.
  - device activation model with max active devices per license/product.
- Community feature maturity:
  - keyword contribution flow exists with private/public states.
  - voting and auto-approval/rejection thresholds implemented.
  - Turnstile + AI moderation + rate limiting integrated.
  - still has fragile areas that should be normalized in Laravel service layer.
- Database model inferred from code (must be migrated with proper Laravel migrations):
  - `emojis`, `emoji_aliases`, `emoji_shortcodes`, `emoji_usage_examples`, `emoji_related`, `emoji_intent_tags`, `emoji_search_tokens`
  - `emoji_keywords`, `keyword_votes`, `moderation_events`
  - `users`
  - `licenses`, `license_activations`, `usage_logs`
  - `ai_guard_logs`, `ai_provider_usage`, `ai_lang_cache`

## Security and configuration migration requirements

- Current live backend keeps many secrets directly in `public_html/config/env.php` (DB, Redis, Turnstile, payment providers, OpenRouter).
- Rebuild must move all secrets to `.env`, rotate exposed credentials, and remove committed secret values from repo history.
- Metrics endpoints currently appear open by default; rebuild should protect admin/internal endpoints with auth or network policy.
- Add internal observability baseline in rebuild:
  - structured request logging
  - protected metrics endpoint(s)
  - deploy healthcheck endpoint

## Important note on Gumroad API tracing

- In `dewemoji-live/public_html/helpers/auth.php`, Gumroad/Mayar validation is currently a stub (`return false`), so live verification logic is not fully present in this folder.
- There is legacy local activation SQL flow in `dewemoji-live/public_html/db.php` (activate/deactivate/verify + device cap), which should be used only as behavioral reference for rebuild design.