dennisthiessenandClaude Opus 5
3e83d63b05
The A5 cutover has been on and observed in production, so SEC Company Facts + DoltHub earnings are already the live source for `fundamental_data`. This removes everything the legacy path still occupied. Gone: the three providers and their config/env keys; the weekly `fundamental_collector` job; the cutover toggle (SEC + Dolt is now the unconditional path, so `off` can no longer silently freeze scoring inputs); the A5 parity report, whose deltas became structurally zero once the candidate builder started writing the table it compared against; and the FMP tier of universe bootstrap. Two behavioral notes: - Disabling **SEC Fundamentals Import** now stops the SEC network fetch only. The local cache refresh moved outside the job-enable check, because candidates also derive from daily closes and earnings events — freezing those on an ingestion pause would stale scoring with no fallback left to recover from. - `/ingestion/fetch?sources=fundamentals` still accepts the key and reports `skipped`; there is no per-ticker fetch any more. Migration 029 does not blanket-delete the leftover settings rows. Migrations run before the service restart, and pre-A6 code reads an absent `job_*_enabled` row as *enabled* — so the two behavior-bearing keys become tombstones pinned to safe values (hidden in Admin) and only the inert three are deleted. Removing the provider keys from the production `.env` is the matching rollout step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
126 lines
4.0 KiB
Bash
Executable File
126 lines
4.0 KiB
Bash
Executable File
#!/usr/bin/env bash
|
||
# Research helpers for a high-CPU MacBook (local only).
|
||
#
|
||
# Kept after Tier-1 cleanup:
|
||
# --ssl-check diagnose corporate CA / proxy
|
||
# --prod-book-matrix re-run 505 vs liquid universe × horizon book matrix
|
||
#
|
||
# Prerequisites: git checkout research branch, .env, deep research.sqlite for
|
||
# book matrix, combined-ca-bundle.pem or certifi when on corp network.
|
||
#
|
||
# chmod +x scripts/run_tier1_macbook.sh
|
||
# ./scripts/run_tier1_macbook.sh --ssl-check
|
||
# ./scripts/run_tier1_macbook.sh --prod-book-matrix
|
||
|
||
set -euo pipefail
|
||
|
||
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||
cd "$ROOT"
|
||
|
||
RESEARCH_SNAP="${RESEARCH_SNAP:-backtest_snapshots/research.sqlite}"
|
||
PROD_SNAP="${PROD_SNAP:-backtest_snapshots/prod.sqlite}"
|
||
WORKERS="${WORKERS:-8}"
|
||
PYTHON="${PYTHON:-python3}"
|
||
USE_CORP_PROXY="${USE_CORP_PROXY:-0}"
|
||
PHASE=""
|
||
|
||
usage() {
|
||
sed -n '2,16p' "$0" | sed 's/^# \?//'
|
||
exit "${1:-0}"
|
||
}
|
||
|
||
while [[ $# -gt 0 ]]; do
|
||
case "$1" in
|
||
--ssl-check) PHASE=ssl; shift ;;
|
||
--prod-book-matrix) PHASE=prod_book; shift ;;
|
||
--corp-proxy) USE_CORP_PROXY=1; shift ;;
|
||
--workers) WORKERS="$2"; shift 2 ;;
|
||
--python) PYTHON="$2"; shift 2 ;;
|
||
-h|--help) usage 0 ;;
|
||
*) echo "Unknown flag: $1" >&2; usage 1 ;;
|
||
esac
|
||
done
|
||
|
||
if [[ -z "$PHASE" ]]; then
|
||
echo "Pick a phase: --ssl-check | --prod-book-matrix" >&2
|
||
usage 1
|
||
fi
|
||
|
||
if [[ -x .venv/bin/python ]]; then
|
||
PYTHON=".venv/bin/python"
|
||
elif command -v "$PYTHON" >/dev/null 2>&1; then
|
||
:
|
||
else
|
||
echo "ERROR: no Python found" >&2
|
||
exit 1
|
||
fi
|
||
|
||
log() { printf '\n==> %s\n' "$*"; }
|
||
die() { echo "ERROR: $*" >&2; exit 1; }
|
||
need_file() { [[ -f "$1" ]] || die "missing $1"; }
|
||
|
||
setup_ssl() {
|
||
export USE_CORP_PROXY
|
||
if [[ -z "${SSL_CERT_FILE:-}" ]]; then
|
||
if [[ -f "$ROOT/combined-ca-bundle.pem" ]]; then
|
||
export SSL_CERT_FILE="$ROOT/combined-ca-bundle.pem"
|
||
elif [[ -f "$HOME/combined-ca-bundle.pem" ]]; then
|
||
export SSL_CERT_FILE="$HOME/combined-ca-bundle.pem"
|
||
fi
|
||
fi
|
||
if [[ -n "${SSL_CERT_FILE:-}" && -f "$SSL_CERT_FILE" ]]; then
|
||
export REQUESTS_CA_BUNDLE="$SSL_CERT_FILE" CURL_CA_BUNDLE="$SSL_CERT_FILE"
|
||
log "SSL CA bundle: $SSL_CERT_FILE"
|
||
else
|
||
local certifi_path
|
||
certifi_path="$("$PYTHON" -c 'import certifi; print(certifi.where())' 2>/dev/null || true)"
|
||
if [[ -n "$certifi_path" && -f "$certifi_path" ]]; then
|
||
export SSL_CERT_FILE="$certifi_path" REQUESTS_CA_BUNDLE="$certifi_path" CURL_CA_BUNDLE="$certifi_path"
|
||
log "SSL CA bundle (certifi): $SSL_CERT_FILE"
|
||
else
|
||
log "WARNING: no CA bundle found — SSL may fail on corp networks"
|
||
fi
|
||
fi
|
||
if [[ "$USE_CORP_PROXY" == "1" ]]; then
|
||
export HTTP_PROXY="${HTTP_PROXY:-http://aproxy.corproot.net:8080}"
|
||
export HTTPS_PROXY="${HTTPS_PROXY:-http://aproxy.corproot.net:8080}"
|
||
export NO_PROXY="${NO_PROXY:-corproot.net,sharedtcs.net,127.0.0.1,localhost}"
|
||
export http_proxy="$HTTP_PROXY" https_proxy="$HTTPS_PROXY" no_proxy="$NO_PROXY"
|
||
log "Corp proxy enabled: $HTTPS_PROXY"
|
||
fi
|
||
export PYTHONPATH="${ROOT}${PYTHONPATH:+:$PYTHONPATH}"
|
||
}
|
||
|
||
ssl_check() {
|
||
setup_ssl
|
||
"$PYTHON" - <<'PY'
|
||
from app.ssl_bootstrap import bootstrap_ssl, ssl_status
|
||
import json, urllib.request
|
||
print(json.dumps(ssl_status(), indent=2))
|
||
print("bootstrap ->", bootstrap_ssl())
|
||
for url in (
|
||
"https://data.alpaca.markets/v2/stocks/SPY/bars?timeframe=1Day&limit=1",
|
||
):
|
||
try:
|
||
req = urllib.request.Request(url, headers={"User-Agent": "ssl-check"})
|
||
with urllib.request.urlopen(req, timeout=20) as resp:
|
||
print(f"OK {resp.status} {url[:60]}")
|
||
except Exception as exc:
|
||
print(f"FAIL {type(exc).__name__}: {exc}")
|
||
PY
|
||
}
|
||
|
||
setup_ssl
|
||
case "$PHASE" in
|
||
ssl) ssl_check ;;
|
||
prod_book)
|
||
need_file "$RESEARCH_SNAP"
|
||
log "Production book universe × horizon matrix"
|
||
"$PYTHON" scripts/run_prod_book_universe_matrix.py \
|
||
--snapshot "$RESEARCH_SNAP" --workers "$WORKERS" --allow-spawn \
|
||
--candidate-cache reports/.cache/prod-book-universe-cands.pkl
|
||
;;
|
||
*) die "unknown phase $PHASE" ;;
|
||
esac
|
||
log "Done."
|