Literate Dotfile Configuration

How a computer is configured... literally(?)!

Welcome to Anton's .dotfiles

Welcome to this org document.
It's mixto writing and code… just some symbols…

It is written in a style known as literate style [Donald Knuth].

This literate .dotfile Org document is
the "single source of truth" for configuratin' me macheens.
Every time I make a change… I give it an explanation here first.
After that, then… the code is tangled.

If yer readin' in a web browser, you'll see only de code.
You're missing the magic links of org.

Each code block you will see from Org mode is manually linked to a specific file.
(a.k.a. tangled -> i.e. automatically inserted into a file)
U gotta read it in emacs to see the magic of org.
One of these lifetimes, emacs will find you.

The tangle of code within these writings… it feels
…as Donald said: [I'm paraphrasing]

Exampleyour code should tell a story

Workflow: How To Use It

Literate Programming

Accepting the practice of writing a literate config means accepting a more complicated workflow.

Since we are authoring a story intertwined with code… as equals…
we have to deliberately place the codes…

This configuration uses literate programming.1

Chezmoi manages the deployed files.2

This applies to the source document.3
It also applies to generated configuration.3

Tangling

Instead of editing our dotfiles directly, we must:
  1. update this document
  2. run a command (tangle) to export that code to the right place

Chezmoi

I've been using chezmoi…
so I already have a similar extra step in the form of chezmoi apply.

Combine Them: Tangle and Apply

However, we'll write a script to help us with these two extra steps.

Run the helper M-x tangle-moi to:
  1. save and tangle, then
  2. chezmoi apply

Config

Below are different sections of the config.

Physical

Direct Input: hands & keyboard

Before installin' n-e-ting… check ya hands an' keys.

Hands

  • touch type: 1.0 accuracy over speed
  • use backspace? walk away and do push/pull-up

Keys

Built-in Mac Keeb

Handled below due to sequential nature of tangling.

Alternative Inputs: voice, face, eyes, body

voice

Talon
os: mac
-
hello talon: "hello world"
select all: key(cmd-a)
copy that: key(cmd-c)
paste that: key(cmd-v)
undo that: key(cmd-z)
redo that: key(cmd-shift-z)

Digital

Chezmoi

First, get chezmoi!

First, We need to:
  1. clone dotfiles repo
  2. run install.sh

Usage:

git clone https://github.com/antoaenono/dotfiles.git ~/.local/share/chezmoi
cd ~/.local/share/chezmoi
./install.sh

install.sh script:

set -e
# Install Homebrew (if not already installed)
if ! command -v brew &>/dev/null; then
    # Download and run the official Homebrew installer
    /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
    # Add brew to PATH for the current session (Apple Silicon path)
    eval "$(/opt/homebrew/bin/brew shellenv)"
fi
# Install chezmoi (dotfile manager)
brew install chezmoi
# Apply dotfiles, run install scripts
chezmoi apply

echo ""
echo "Done. Restart your computer to ensure everything is loaded correctly."
Notes on the Homebrew install:

Configuration

[scriptEnv]
    VAR_IN_SCRIPT = "a_value"

Repository-only files are excluded from the target home directory.

**/README.*
config.org
install.sh
Brewfile
AGENTS.md
.codex/**

# Skip macOS-only scripts on non-macOS systems
{{ if ne .chezmoi.os "darwin" }}
.chezmoiscripts/darwin/**
{{ end }}

macOS

Homebrew Packages via Brewfile

This script embeds the Brewfile SHA-256 hash in a comment.
This causes chezmoi to rerun brew bundle on the next chezmoi apply
if the Brewfile content has changed.4

set -e
# Brewfile hash: {{ include "Brewfile" | sha256sum }}
brew bundle --no-upgrade --file="{{ joinPath .chezmoi.sourceDir "Brewfile" }}"

Shell (Zsh)

oh-my-zsh
set -e
if [ ! -d "$HOME/.oh-my-zsh" ]; then
    git clone --depth=1 https://github.com/ohmyzsh/ohmyzsh.git "$HOME/.oh-my-zsh"
fi
Starship
brew "starship"

Starship init is handled by the oh-my-zsh starship plugin (see plugins in .zshrc).

Aliases
alias h='history'
alias l='ls -la'
Functions
# Create and then navigate to new dir
mkd() {
  mkdir -p "$1" && cd "$1"
}
Zsh Configuration Files

These are documented in order they are loaded.

zshenv

Every shell (login, interactive, script). Loaded first, always.
The ~/.zshenv is a symlink to ~/.config/zsh/.zshenv, which bootstraps ZDOTDIR so zsh finds the rest of its config in ~/.config/zsh/.
In other words, because we're using the XDG_CONFIG_HOME standard (~/.config directory), we need to symlink .zshenv which MUST be in the home directory.

Symlink:

{{ .chezmoi.homeDir }}/.config/zsh/.zshenv
export XDG_CONFIG_HOME=${XDG_CONFIG_HOME:=${HOME}/.config}
export ZDOTDIR=${ZDOTDIR:=${XDG_CONFIG_HOME}/zsh}
zprofile

Login shells only, before .zshrc. Runs once at login, not every new tab.

# Set PATH for Homebrew
eval "$(/opt/homebrew/bin/brew shellenv)"
zshrc

Interactive shells only. Every new terminal tab.
This is where the bulk of shell config lives.

# oh-my-zsh
export ZSH="$ZDOTDIR/ohmyzsh"
zstyle ':omz:update' mode reminder
plugins=(git starship)
source $ZSH/oh-my-zsh.sh

# Aliases
source ~/.config/zsh/alias.zsh

# Functions
source ~/.config/zsh/function.zsh
zlogin

Login shells only, after .zshrc. Not currently used.

zlogout

On shell exit. Not currently used.

Core CLI tools

brew "coreutils"
brew "fd"
brew "ripgrep"
brew "wget"

AeroSpace

Tiling window manager

tap "nikitabobko/tap", trusted: { cask: "aerospace" }
cask "nikitabobko/tap/aerospace"
Commands

All commands: https://nikitabobko.github.io/AeroSpace/commands (opens in a new tab)

Run commands from:
  1. terminal (prefix with aerospace)
  2. key binding

If shell completion is working, try aerospace TAB to see all commands.

Example: open terminal with alt + enter shortcut (like in i3)
https://nikitabobko.github.io/AeroSpace/commands#exec-and-forget (opens in a new tab)

Examplealt-enter = '''exec-and-forget osascript -e '
tell application "Terminal"
    do script
    activate
end tell'
'''
Startup
start-at-login = true

# https://github.com/FelixKratz/JankyBorders
after-startup-command = [
  'exec-and-forget borders active_color=0xff7dcfff inactive_color=0xff494d64 width=10.0'
]
set -e
/usr/bin/defaults write bobko.aerospace displayStyle -string squares
Properties
# Focus: mouse follow
#on-focused-monitor-changed = ['move-mouse monitor-lazy-center'] # follow monitor
on-focus-changed = ['move-mouse window-lazy-center'] # follow window
# https://nikitabobko.github.io/AeroSpace/commands#move-mouse
# https://nikitabobko.github.io/AeroSpace/guide#on-focus-changed-callbacks

# Window: layouts
# https://nikitabobko.github.io/AeroSpace/guide#layouts
default-root-container-layout = 'tiles' # tiles|accordion
default-root-container-orientation = 'auto' # horizontal|vertical|auto
# auto: wide monitor => horizontal, tall monitor => vertical
accordion-padding = 42

# Window: tree structure for containers: normalization property
# https://nikitabobko.github.io/AeroSpace/guide#normalization
enable-normalization-flatten-containers = true
enable-normalization-opposite-orientation-for-nested-containers = true

# Misc:
# Toggle off macOS "Hide application", useful if accidentally hit cmd-h
# https://nikitabobko.github.io/AeroSpace/goodies#disable-hide-app
automatically-unhide-macos-hidden-apps = true
Key-mapping
  • don't think I need this any more…
  • not sure what it's for since we override bindings anyway
# https://nikitabobko.github.io/AeroSpace/guide#key-mapping
# [key-mapping]
#     preset = 'qwerty' # qwerty|dvorak|colemak

Gaps

# Gaps between windows (inner-*) and between monitor edges (outer-*).
# Possible values:
# - Constant:     gaps.outer.top = 8
# - Per monitor:  gaps.outer.top = [{ monitor.main = 16 }, { monitor."some-pattern" = 32 }, 24]
#                 In this example, 24 is a default value when there is no match.
#                 Monitor pattern is the same as for 'workspace-to-monitor-force-assignment'.
#                 See:
#                 https://nikitabobko.github.io/AeroSpace/guide#assign-workspaces-to-monitors
[gaps]
    inner.horizontal = 2
    inner.vertical =   0

    outer.left =       5
    outer.bottom =     5
    outer.top =        5
    outer.right =      5
Key Bindings
Key Names
  • Letters. a, b, c, …, z
  • Numbers. 0, 1, 2, …, 9
  • Keypad numbers. keypad0, keypad1, keypad2, …, keypad9
  • F-keys. f1, f2, …, f20
  • Special keys. minus, equal, period, comma, slash, backslash, quote, semicolon,
    backtick, leftSquareBracket, rightSquareBracket, space, enter, esc,
    backspace, tab, pageUp, pageDown, home, end, forwardDelete,
    sectionSign (ISO keyboards only, european keyboards only)
  • Keypad special. keypadClear, keypadDecimalMark, keypadDivide, keypadEnter, keypadEqual,
    keypadMinus, keypadMultiply, keypadPlus
  • Arrows. left, down, up, right
  • Modifiers. cmd, alt, ctrl, shift
Main Mode
[mode.main.binding]
    # Modes  https://nikitabobko.github.io/AeroSpace/commands#mode
    # You can make up any kind of mode you want!
    # https://nikitabobko.github.io/AeroSpace/guide#binding-modes
    alt-cmd-semicolon = 'mode service'

    # Layout https://nikitabobko.github.io/AeroSpace/commands#layout
    alt-cmd-period= 'layout accordion vertical horizontal'
    alt-cmd-slash = 'layout tiles horizontal vertical'

    # Resize https://nikitabobko.github.io/AeroSpace/commands#resize
    alt-cmd-minus = 'resize smart -150'
    alt-cmd-equal = 'resize smart +150'

    # Focus https://nikitabobko.github.io/AeroSpace/commands#focus
    alt-cmd-h = 'focus left'
    alt-cmd-left = 'focus left'
    alt-cmd-j = 'focus down'
    alt-cmd-down = 'focus down'
    alt-cmd-k = 'focus up'
    alt-cmd-up= 'focus up'
    alt-cmd-l = 'focus right'
    alt-cmd-right = 'focus right'

    # Move https://nikitabobko.github.io/AeroSpace/commands#move
    ctrl-alt-cmd-h = 'move left'
    ctrl-alt-cmd-left = 'move left'
    ctrl-alt-cmd-j = 'move down'
    ctrl-alt-cmd-down = 'move down'
    ctrl-alt-cmd-k = 'move up'
    ctrl-alt-cmd-up = 'move up'
    ctrl-alt-cmd-l = 'move right'
    ctrl-alt-cmd-right = 'move right'

    # Workspace recent https://nikitabobko.github.io/AeroSpace/commands#workspace-back-and-forth
    alt-cmd-tab = 'workspace-back-and-forth'
    # Workspace monitor https://nikitabobko.github.io/AeroSpace/commands#move-workspace-to-monitor
    ctrl-alt-cmd-tab = 'move-workspace-to-monitor --wrap-around next'

    # Workspace show https://nikitabobko.github.io/AeroSpace/commands#workspace
    alt-cmd-1 = 'workspace 1'
    alt-cmd-2 = 'workspace 2'
    alt-cmd-3 = 'workspace 3'
    alt-cmd-4 = 'workspace 4'
    alt-cmd-5 = 'workspace 5'
    alt-cmd-6 = 'workspace 6'
    alt-cmd-7 = 'workspace 7'
    alt-cmd-8 = 'workspace 8'
    alt-cmd-9 = 'workspace 9'
    alt-cmd-a = 'workspace A'
    alt-cmd-b = 'workspace B'
    alt-cmd-c = 'workspace C'
    alt-cmd-d = 'workspace D'
    alt-cmd-e = 'workspace E'
    alt-cmd-f = 'workspace F'
    alt-cmd-g = 'workspace G'
    alt-cmd-i = 'workspace I'
    alt-cmd-m = 'workspace M'
    alt-cmd-n = 'workspace N'
    alt-cmd-o = 'workspace O'
    alt-cmd-p = 'workspace P'
    alt-cmd-q = 'workspace Q'
    alt-cmd-r = 'workspace R'
    alt-cmd-s = 'workspace S'
    alt-cmd-t = 'workspace T'
    alt-cmd-u = 'workspace U'
    alt-cmd-w = 'workspace W'
    alt-cmd-x = 'workspace X'
    alt-cmd-y = 'workspace Y'
    alt-cmd-z = 'workspace Z'

    # Workspace move and follow https://nikitabobko.github.io/AeroSpace/commands#move-node-to-workspace
    ctrl-alt-cmd-1 = 'move-node-to-workspace --focus-follows-window 1'
    ctrl-alt-cmd-2 = 'move-node-to-workspace --focus-follows-window 2'
    ctrl-alt-cmd-3 = 'move-node-to-workspace --focus-follows-window 3'
    ctrl-alt-cmd-4 = 'move-node-to-workspace --focus-follows-window 4'
    ctrl-alt-cmd-5 = 'move-node-to-workspace --focus-follows-window 5'
    ctrl-alt-cmd-6 = 'move-node-to-workspace --focus-follows-window 6'
    ctrl-alt-cmd-7 = 'move-node-to-workspace --focus-follows-window 7'
    ctrl-alt-cmd-8 = 'move-node-to-workspace --focus-follows-window 8'
    ctrl-alt-cmd-9 = 'move-node-to-workspace --focus-follows-window 9'
    ctrl-alt-cmd-a = 'move-node-to-workspace --focus-follows-window A'
    ctrl-alt-cmd-b = 'move-node-to-workspace --focus-follows-window B'
    ctrl-alt-cmd-c = 'move-node-to-workspace --focus-follows-window C'
    ctrl-alt-cmd-d = 'move-node-to-workspace --focus-follows-window D'
    ctrl-alt-cmd-e = 'move-node-to-workspace --focus-follows-window E'
    ctrl-alt-cmd-f = 'move-node-to-workspace --focus-follows-window F'
    ctrl-alt-cmd-g = 'move-node-to-workspace --focus-follows-window G'
    ctrl-alt-cmd-i = 'move-node-to-workspace --focus-follows-window I'
    ctrl-alt-cmd-m = 'move-node-to-workspace --focus-follows-window M'
    ctrl-alt-cmd-n = 'move-node-to-workspace --focus-follows-window N'
    ctrl-alt-cmd-o = 'move-node-to-workspace --focus-follows-window O'
    ctrl-alt-cmd-p = 'move-node-to-workspace --focus-follows-window P'
    ctrl-alt-cmd-q = 'move-node-to-workspace --focus-follows-window Q'
    ctrl-alt-cmd-r = 'move-node-to-workspace --focus-follows-window R'
    ctrl-alt-cmd-s = 'move-node-to-workspace --focus-follows-window S'
    ctrl-alt-cmd-t = 'move-node-to-workspace --focus-follows-window T'
    ctrl-alt-cmd-u = 'move-node-to-workspace --focus-follows-window U'
    ctrl-alt-cmd-w = 'move-node-to-workspace --focus-follows-window W'
    ctrl-alt-cmd-x = 'move-node-to-workspace --focus-follows-window X'
    ctrl-alt-cmd-y = 'move-node-to-workspace --focus-follows-window Y'
    ctrl-alt-cmd-z = 'move-node-to-workspace --focus-follows-window Z'
Service Mode
[mode.service.binding]
    esc = ['reload-config', 'mode main']

    # Window float toggle
    f = ['layout floating tiling', 'mode main']

    # r = ['flatten-workspace-tree', 'mode main'] # reset layout
    # backspace = ['close-all-windows-but-current', 'mode main']

    # sticky is not yet supported https://github.com/nikitabobko/AeroSpace/issues/2
    # s = ['layout sticky tiling', 'mode main']

    # alt-shift-h = ['join-with left', 'mode main']
    # alt-shift-j = ['join-with down', 'mode main']
    # alt-shift-k = ['join-with up', 'mode main']
    # alt-shift-l = ['join-with right', 'mode main']

    # down = 'volume down'
    # up = 'volume up'
    # shift-down = ['volume set 0', 'mode main']
Automations
# To find an `app-id`, use `aerospace list-apps`
# https://nikitabobko.github.io/AeroSpace/commands#list-apps

# Automatically run functions for specific windows
# https://nikitabobko.github.io/AeroSpace/guide#on-window-detected-callback

# Only `move-node-to-workspace`, `layout tiling`, `layout floating` are supported
# https://github.com/nikitabobko/AeroSpace/issues/20
# this limitation is supposed to be removed in 0.2
# https://github.com/nikitabobko/AeroSpace/milestone/5
[[on-window-detected]]
  if.app-id = 'com.apple.ActivityMonitor'
  run = ['move-node-to-workspace A']

[[on-window-detected]]
  if.app-id = 'org.gnu.Emacs'
  run = ['move-node-to-workspace E']

[[on-window-detected]]
  if.app-id = 'com.google.Chrome'
  run = ['move-node-to-workspace C']

[[on-window-detected]]
  if.app-id = 'com.apple.finder'
  run = ['move-node-to-workspace F']

[[on-window-detected]]
  if.app-id = 'com.apple.Safari'
  run = ['move-node-to-workspace S']

[[on-window-detected]]
  if.app-id = 'com.mitchellh.ghostty'
  run = ['move-node-to-workspace T']

[[on-window-detected]]
  if.app-id = 'com.openai.codex'
  run = ['move-node-to-workspace X']

[[on-window-detected]]
  if.app-id = 'local.codex.viewer'
  run = ['move-node-to-workspace X']

[[on-window-detected]]
  if.app-id = 'dev.zed.Zed'
  run = ['move-node-to-workspace Z']

BitWarden

SSH agent

To defer to Bitwarden's SSH agent on `commit` and `push`,
this environment variable must be set:

# Bitwarden SSH Agent
export SSH_AUTH_SOCK=$HOME/Library/Containers/com.bitwarden.desktop/Data/.bitwarden-ssh-agent.sock

LaTeX

cask "mactex"

After a major mactex upgrade, TeX Live regenerates format files (opens in a new tab) (fmtutil) and font map files (updmap) on first run. This blocks any process that calls pdflatex synchronously (including Emacs during org export). See TeX Live Guide §3.4.4: System font configuration (opens in a new tab) and updmap/fmtutil sys vs user (opens in a new tab) for background. The script below warms the cache after each mactex upgrade.

# mactex version: {{ output "brew" "list" "--cask" "--versions" "mactex" | trim }}
# Warms TeX Live font and format caches on first run after install or upgrade.
# Without this, the first pdflatex call blocks synchronously while regenerating
# caches, which freezes any process waiting on it (e.g. Emacs during org export).
# Only warms pdflatex; to warm all engines (xelatex, lualatex, etc.) use instead:
#   fmtutil-user --all && updmap-user  (~90s)
set -e
pdflatex -interaction=nonstopmode /dev/null 2>/dev/null || true

Emacs

This standalone Mac application (opens in a new tab) is well maintained.

tap "jimeh/emacs-builds", trusted: { cask: "emacs-app" }
cask "jimeh/emacs-builds/emacs-app"
Starting Emacs

--alternate-editor""= is a historical hack: passing empty string triggers Emacs to start a daemon as the fallback if no server is running.

# ec: open emacsclient frame, detached (-n), starting daemon if needed
alias ec='emacsclient -c -n --alternate-editor=""'
# ecc: interactive emacsclient - reports server status, then opens a frame
ecs() {
  if [[ "$1" == "-k" ]]; then
    if ! emacsclient -e t &>/dev/null; then
      echo "no server running"
      return 1
    fi
    read "reply?kill server? [y/n] "
    if [[ "$reply" == "y" ]]; then
      emacsclient -e '(kill-emacs)'
      echo "server killed"
    else
      echo "aborted"
    fi
    return
  fi
  if emacsclient -e t &>/dev/null; then
    echo "server: running"
  else
    echo "server: not found - starting daemon..."
    emacs --daemon
  fi
  local server_name
  server_name=$(emacsclient -e 'server-name' | tr -d '"')
  echo "--- emacs state ---"
  emacs --batch --eval "(progn (require 'server) (princ (server-eval-at \"$server_name\" '(let* ((frames (seq-filter (lambda (f) (frame-parameter f 'client)) (frame-list))) (lines (seq-map-indexed (lambda (f i) (format \"frame %d:\\n  workspace: %s\\n  windows (active buffer): %s\" (1+ i) (with-selected-frame f (persp-name (get-current-persp))) (string-join (mapcar (lambda (w) (buffer-name (window-buffer w))) (window-list f)) \", \"))) frames))) (string-join lines \"\\n\")))))" 2>/dev/null
  emacsclient -c -n
  echo "\n\nnew frame opened"
}
Doom
Install

Download

set -e
xcode-select -p &>/dev/null || xcode-select --install
if [ ! -d "$HOME/.emacs.d" ]; then
    git clone --depth=1 https://github.com/doomemacs/doomemacs "$HOME/.emacs.d"
    "$HOME/.emacs.d/bin/doom" install
fi

Add doom command to PATH

# doom
export PATH="$HOME/.emacs.d/bin:$PATH"
Configuration
tangle-moi.el

Portable Emacs implementation shared by the interactive command and agent
launcher.

;;; tangle-moi.el --- Tangle the literate dotfiles config -*- lexical-binding: t; -*-

(require 'ob-tangle)
(require 'subr-x)

(defun tangle-moi--source-directory ()
  "Return the chezmoi source directory."
  (file-name-as-directory
   (string-trim (car (process-lines "chezmoi" "source-path")))))

(defun tangle-moi--config-file ()
  "Return the canonical config.org path."
  (file-truename
   (expand-file-name "config.org" (tangle-moi--source-directory))))

(defun tangle-moi-tangle ()
  "Tangle config.org while preserving any unsaved live Emacs buffer."
  (let* ((source-directory (tangle-moi--source-directory))
         (config-file (tangle-moi--config-file))
         (existing-buffer (find-buffer-visiting config-file)))
    (when (and existing-buffer (buffer-modified-p existing-buffer))
      (user-error "Refusing to tangle: config.org has unsaved Emacs changes"))
    (with-current-buffer (or existing-buffer (find-file-noselect config-file))
      (unless (verify-visited-file-modtime (current-buffer))
        (revert-buffer t t))
      (let ((default-directory source-directory))
        (save-restriction
          (widen)
          (org-babel-tangle))))))

(defun tangle-moi-tangle-batch ()
  "Tangle config.org and print the target files for batch callers."
  (condition-case error-data
      (progn
        (prin1 (tangle-moi-tangle))
        (terpri))
    (error
     (message "Tangle failed: %s" (error-message-string error-data))
     (kill-emacs 1))))

(defun tangle-moi--save-config-buffer ()
  "Save config.org when it has unsaved changes in Emacs."
  (when-let ((buffer (find-buffer-visiting (tangle-moi--config-file))))
    (when (buffer-modified-p buffer)
      (with-current-buffer buffer
        (save-buffer)))))

;;;###autoload
(defun tangle-moi ()
  "Tangle the literate dotfiles config, preview it, and ask to apply."
  (interactive)
  (tangle-moi--save-config-buffer)
  (message "Tangling Org file...")
  (tangle-moi-tangle)
  (message "Tangling complete. Previewing chezmoi changes...")
  (let ((diff (shell-command-to-string
               "chezmoi apply --dry-run --verbose --force")))
    (if (string-empty-p diff)
        (message "No changes to apply.")
      (with-output-to-temp-buffer "*chezmoi diff*"
        (princ diff))
      (if (y-or-n-p
           "Apply these changes, overwriting any drifted targets? ")
          (if (zerop
               (shell-command "chezmoi apply --verbose --force"))
              (message "Changes applied.")
            (message "Chezmoi apply failed; see *Shell Command Output*."))
        (message "Changes aborted.")))))

(provide 'tangle-moi)
;;; tangle-moi.el ends here
config.el
;;; $DOOMDIR/config.el -*- lexical-binding: t; -*-

; Remember, you do not need to run 'doom sync' after modifying this file!

;; Some functionality uses this to identify you, e.g. GPG configuration, email
;; clients, file templates and snippets. It is optional.
;; (setq user-full-name "John Doe"
;;       user-mail-address "john@doe.com")

;; Doom exposes five (optional) variables for controlling fonts in Doom:
;;
;; - `doom-font' -- the primary font to use
;; - `doom-variable-pitch-font' -- a non-monospace font (where applicable)
;; - `doom-big-font' -- used for `doom-big-font-mode'; use this for
;;   presentations or streaming.
;; - `doom-symbol-font' -- for symbols
;; - `doom-serif-font' -- for the `fixed-pitch-serif' face
;;
;; See 'C-h v doom-font' for documentation and more examples of what they
;; accept. For example:
;;
;;(setq doom-font (font-spec :family "Fira Code" :size 12 :weight 'semi-light)
;;      doom-variable-pitch-font (font-spec :family "Fira Sans" :size 13))
;;
;; If you or Emacs can't find your font, use 'M-x describe-font' to look them
;; up, `M-x eval-region' to execute elisp code, and 'M-x doom/reload-font' to
;; refresh your font settings. If Emacs still can't find your font, it likely
;; wasn't installed correctly. Font issues are rarely Doom issues!

(setopt
  user-full-name "Anton Bilbaeno"
  user-mail-address "antoaenono@pm.me"
  display-line-numbers-type 'relative
  mouse-drag-copy-region t
)

(setq
  doom-theme 'doom-monokai-machine
)

(setq
  projectile-project-search-path '("~/source/" "~/.local/share/chezmoi/")
  projectile-globally-ignored-directories '("node_modules" ".cache" "deps" "_build" "dist" ".elixir_ls" ".hex" ".mix")
  projectile-ignored-projects '("~/source/archive/")
)

(load! "lisp/tangle-moi")


;; 80 char limit encourages concise code. Respect to the early days.
;; note: in org files 80 chars is /on top/ of the spacing under the heading
(setq-default fill-column 80)
(global-display-fill-column-indicator-mode 1)

;; Rebind "quit" to Cmd-Shift-q to avoid conflict with M-q (fill-paragraph)
;; ...shouldn't be so easy to quit anyway!
(map! :g "M-Q" #'save-buffers-kill-terminal)
;; TODO the above doesn't work, it just adds a new binding, and Emacs Gui still captures Cmd+q


;; HASKELL
;; Hooks so haskell and literate haskell major modes trigger LSP setup
(add-hook 'haskell-mode-hook #'lsp)
(add-hook 'haskell-literate-mode-hook #'lsp)

;; TODO test if these are really needed after moving path to .zshenv (was in .zshrc)
;; Add haskell lsp to path for emacs subprocesses
(add-to-list 'exec-path (expand-file-name "~/.ghcup/bin"))
;; Add haskell tools to path for emacs environment
(setenv "PATH" (concat (expand-file-name "~/.ghcup/bin") ":" (getenv "PATH")))
;; hlint in local bin. Ensure ~/.local/bin is in Emacs's exec-path
;; TODO this didnt work emacs still can't find hlint
(setq exec-path (cons (expand-file-name "~/.local/bin") exec-path))

;; Counted normal-state j/k match relative buffer-line numbers across folds.
;; Uncounted motions and operator/visual-state bindings retain Evil's defaults.
(defcustom my/counted-motions-reveal nil
  "Whether counted j/k reveal a destination inside folded text."
  :type 'boolean
  :group 'evil)

(after! evil
  (defun my/move-buffer-lines (count)
    "Move COUNT buffer lines, resolving hidden destinations without guessing."
    (let ((line-move-visual nil)
          (line-move-ignore-invisible nil))
      (evil-line-move count))
    (when (invisible-p (point))
      (if my/counted-motions-reveal
          (cond ((derived-mode-p 'org-mode) (org-fold-reveal))
                ((derived-mode-p 'markdown-mode) (markdown-show-entry))
                (t (+fold/open)))
        ;; Leave folds closed and stop at the boundary in the motion direction.
        (while (and (invisible-p (point))
                    (if (> count 0) (not (eobp)) (not (bobp))))
          (goto-char
           (if (> count 0)
               (next-char-property-change (point) (point-max))
             (max (point-min)
                  (1- (previous-char-property-change (point) (point-min)))))))
        ;; Org can leave a visible newline at the end of a hidden span.
        (when (and (> count 0) (eolp) (not (eobp)))
          (forward-char 1))))
    (evil-adjust-cursor))

  (evil-define-motion my/next-line (count)
    "Move by buffer lines with COUNT and relative numbers, otherwise normally."
    :type line
    (if (and count (eq display-line-numbers 'relative))
        (my/move-buffer-lines count)
      (evil-next-line count)))

  (evil-define-motion my/previous-line (count)
    "Move up by buffer lines with COUNT and relative numbers, otherwise normally."
    :type line
    (if (and count (eq display-line-numbers 'relative))
        (my/move-buffer-lines (- count))
      (evil-previous-line count)))

  (define-key evil-normal-state-map "j" #'my/next-line)
  (define-key evil-normal-state-map "k" #'my/previous-line))

;; ORG ;;
(setopt org-directory "~/org/")

;; Restore Org's native folded -> children -> subtree TAB cycle. Doom's
;; evil-org integration otherwise replaces it with a two-state local toggle.
(after! evil-org
  (remove-hook 'org-tab-first-hook #'+org-cycle-only-current-subtree-h))

;; Disable org-element cache (Org 9.5+ bug): subtree and narrowed exports trigger
;; cache revalidation on the temp export buffer, causing an infinite loop / hang.
;; Full-document export is unaffected. Disabling the cache trades minor parse speed
;; for reliable export at any scope.
(setq org-element-use-cache nil)

;; (after! org
;;   ;; Add the 'titletoc' package to Org's list of LaTeX packages
;;   (add-to-list 'org-latex-packages-alist '("titletoc" nil t))

;;   ;; Use latexmk for PDF export
;;   ;; (setq org-latex-to-pdf-process (list "latexmk %f"))
;; )

;; more `<` tab completion templates aka "structure templates" -- orgmode.org/manual/Structure-Templates.html
;(after! org (add-to-list 'org-structure-template-alist
  ;("sh" . "src shell")))

;; custom syntax: !! to indicate key presses
;; add as a keyword in org-mode and draw a box around it
;; TODO export to <kbd>
(after! org
  (font-lock-add-keywords
   'org-mode
   '(("!!\\(.*?\\)!!"
      (1 '(face (:weight 'semi-bold
                           :box '(:line-width 2 :color "palegoldenrod"))))))))

;; automatically show magic behind point
(use-package! org-appear
  :after org
  :hook (org-mode . org-appear-mode)
  :config
  (setopt org-appear-autoemphasis t ; _/**/_
          org-appear-autolinks t ;; dun work on #+TRANSCLUSION directives. use `org-toggle-link-display`
          org-appear-autoentities t ;; \alpha for α
  )
)

; information teleport
(use-package! org-transclusion
  :after org
  :init
  (map!
   :map global-map "<f12>" #'org-transclusion-add
   :leader
   :prefix "n"
   :desc "Org Transclusion Mode" "t" #'org-transclusion-mode))

; https://nobiot.github.io/org-transclusion/#:~:text=4.10%20Extensions,regain%20the%20indentation.
; this didn't seem to work, or i don't understand what it's supposed to be doing
; but i was trying to fix the issue where the source buffer loses indentation while transclusion is active
  ; :config
  ; ;; Enable org-transclusion-indent-mode extension
  ; (add-to-list 'org-transclusion-extensions 'org-transclusion-indent-mode)
  ; (require 'org-transclusion-indent-mode))

; roaman emacs... to gather
(setopt org-roam-directory "~/org/roam")
(org-roam-db-autosync-mode) ; docs say put this here[?]

;; CITATIONS (citar + org-cite)
(setq! citar-bibliography '("~/org/bib/references.bib")
       citar-library-paths '("~/org/pdfs/")
       citar-notes-paths '("~/org/roam/refs/"))

(after! oc
  (setq org-cite-global-bibliography '("~/org/bib/references.bib")
        org-cite-insert-processor 'citar
        org-cite-follow-processor 'citar
        org-cite-activate-processor 'citar))

;; bridge citar with org-roam for note management
(use-package! citar-org-roam
  :after (citar org-roam)
  :config
  (citar-org-roam-mode)
  (setq citar-org-roam-subdir "refs"
        citar-org-roam-note-title-template "${author}, ${title}"
        citar-org-roam-template-fields
        '((:citar-title   . ("title"))
          (:citar-author  . ("author" "editor"))
          (:citar-date    . ("date" "year" "issued"))
          (:citar-pages   . ("pages"))
          (:citar-type    . ("=type="))
          (:citar-file    . ("file"))))
  (add-to-list 'org-roam-capture-templates
               '("r" "reference" plain "%?"
                 :if-new (file+head "refs/${citar-citekey}.org"
                                    ":PROPERTIES:\n:NOTER_DOCUMENT: ${citar-file}\n:END:\n#+title: ${citar-title}\n#+DATE: %<%Y-%m-%d %a>\n#+PUBLISH: garden\n")
                 :immediate-finish t
                 :unnarrowed t))
  (setq citar-org-roam-capture-template-key "r"))

(defun my/citar-open-noter ()
  "Open roam note and start org-noter in a dedicated frame."
  (interactive)
  (select-frame (make-frame))
  (call-interactively #'citar-open-notes)
  (org-noter))

(map! :leader :desc "Open noter session" "n p" #'my/citar-open-noter)

(defun my/citar-close-noter ()
  "Save notes and PDF annotations, end org-noter session, and close frame."
  (interactive)
  (org-noter--with-valid-session
    (with-current-buffer (org-noter--session-notes-buffer session)
      (save-buffer))
    (with-current-buffer (org-noter--session-doc-buffer session)
      (save-buffer)))
  (org-noter-kill-session)
  (delete-frame))

(map! :leader :desc "Close noter session" "n q" #'my/citar-close-noter)

;; PDF ANNOTATION (org-noter)
(use-package! org-noter
  :after org
  :config
  (setq org-noter-always-create-frame nil
        org-noter-kill-frame-at-session-end nil
        org-noter-auto-save-last-location nil
        org-noter-doc-property-in-notes t
        org-noter-notes-window-location 'horizontal-split
        org-noter-highlight-selected-text t))

;; GLOSSARY / ACRONYMS
(setq org-link-descriptive t
      org-glossary-global-terms '("~/org/glossary.org"))
(use-package! org-glossary
  :after org
  :hook (org-mode . org-glossary-mode)
  :config
  (load! "lisp/org-glossary-export-fix"))

;; PEOPLE REFERENCES
(use-package! org-contacts
  :after org
  :config
  (setq org-contacts-files '("~/org/contacts.org")
        org-contacts-matcher "contact"
        org-contacts-tag "contact")
  (add-to-list 'org-tags-exclude-from-inheritance "contact")

  (defun my/org-contact-description (link _description)
    "Default LINK's local description to its nickname or initials."
    (let* ((name (org-link-unescape
                  (replace-regexp-in-string "\\`\\(?:org-contact\\|who\\):" "" link)))
           (contact (car (org-contacts-filter
                          (concat "\\`" (regexp-quote name) "\\'"))))
           (nickname (cdr (assoc-string "NICKNAME" (nth 2 contact)))))
      (if (and nickname (not (string-empty-p (string-trim nickname))))
          (string-trim nickname)
        (mapconcat (lambda (word) (upcase (substring word 0 1)))
                   (split-string name "[[:space:]-]+" t) ""))))

  (org-link-set-parameters "org-contact"
                           :insert-description #'my/org-contact-description)

  (defun my/org-contact-short-complete (&optional arg)
    "Complete a who: link using org-contacts."
    (concat "who:" (string-remove-prefix "org-contact:"
                                         (org-contacts-link-complete arg))))

  (org-link-set-parameters "who"
                           :follow #'org-contacts-link-open
                           :complete #'my/org-contact-short-complete
                           :insert-description #'my/org-contact-description)

  (require 'url-util)

  (defun my/org-contact-social-url (property value)
    "Resolve a GitHub or LinkedIn PROPERTY VALUE to its profile URL."
    (when-let ((template (cdr (assoc-string
                              property
                              '(("GITHUB" . "https://github.com/%s")
                                ("LINKEDIN" . "https://www.linkedin.com/in/%s/"))
                              t))))
      (let* ((value (string-trim value))
             (handle (string-remove-prefix "@" value)))
        (cond ((string-match-p "\\`https?://[^[:space:]]+\\'" value) value)
              ((string-match-p "\\`[[:alnum:]][[:alnum:]._-]*\\'" handle)
               (format template (url-hexify-string handle)))))))

  (defun my/org-open-contact-social-property (&rest _)
    "Open a GitHub or LinkedIn profile from the Org property at point."
    (when (derived-mode-p 'org-mode)
      (let ((property (org-element-context)))
        (when (eq (org-element-type property) 'node-property)
          (when-let ((url (my/org-contact-social-url
                          (org-element-property :key property)
                          (org-element-property :value property))))
            (browse-url url)
            t)))))

  (add-hook 'org-open-at-point-functions #'my/org-open-contact-social-property)
  (advice-add '+org/dwim-at-point :before-until #'my/org-open-contact-social-property))

(defun my/org-insert-contact-link ()
  "Select a contact using org-contacts and insert its default label."
  (interactive)
  (require 'org-contacts)
  (org-insert-link nil (my/org-contact-short-complete)))

(map! :leader :desc "Insert contact reference" "n P" #'my/org-insert-contact-link)

;; ORG HEADING FACES (levels 5-8 inherit outline-4/foreground by default)
(custom-set-faces!
  '(org-level-5 :foreground "#f92672")  ; pink
  '(org-level-6 :foreground "#fd971f")  ; orange
  '(org-level-7 :foreground "#ae81ff")  ; purple
  '(org-level-8 :foreground "#66d9ef")) ; teal

;; ROAM UI (graph visualization)
(use-package! org-roam-ui
  :after org-roam
  :config
  (setq org-roam-ui-sync-theme t
        org-roam-ui-follow t
        org-roam-ui-update-on-save t))

;; CAPTURE TEMPLATES
(after! org
  (setq org-capture-templates
        '(("t" "Task" entry (file+headline "~/org/inbox.org" "Tasks")
           "* TODO %?\n%U\n%a\n")
          ("n" "Note" entry (file+headline "~/org/inbox.org" "Notes")
           "* %?\n%U\n")
          ("r" "Reading Note" entry (file+headline "~/org/inbox.org" "Reading")
           "* %^{Title}\n%U\n%?\n")
          ("p" "Person" entry (file "~/org/contacts.org")
           "* %^{Name} :contact:\n:PROPERTIES:\n:ID: %(org-id-new)\n:NICKNAME: %^{Nickname}\n:PRIVATE: true\n:END:\n%?\n"))))

;; REFILE
(after! org
  (setq org-refile-targets '((nil :maxlevel . 3)
                             (org-agenda-files :maxlevel . 3))
        org-refile-use-outline-path 'file
        org-outline-path-complete-in-steps nil
        org-refile-allow-creating-parent-nodes 'confirm))

;; AGENDA (recursive - finds TODOs in all subdirs)
(setq org-agenda-files '("~/org/")
      org-agenda-file-regexp "\\`[^.].*\\.org\\'")
(setq org-agenda-custom-commands
      '(("d" "Dashboard"
         ((agenda "" ((org-agenda-span 14)))
          (todo "TODO" ((org-agenda-overriding-header "Tasks")))))))

(defun my/org-agenda-files-recursive ()
  "Build agenda file list recursively from ~/org/."
  (directory-files-recursively "~/org/" "\\.org$"))
(setq org-agenda-files (my/org-agenda-files-recursive))

(defun my/insert-site-file-header (publish-type)
  "Insert a dated draft site header routed to PUBLISH-TYPE."
  (let ((title
         (capitalize
          (string-replace
           "-" " "
           (file-name-base buffer-file-name)))))
    (insert
     (format "#+title: %s\n#+DATE: %s\n#+PUBLISH: %s\n#+DRAFT: true\n\n"
             title
             (format-time-string "<%Y-%m-%d %a>")
             publish-type))))

;; Publishing is metadata-driven, so source location and site destination can
;; vary independently. These rules provide the common directory defaults.
(set-file-template!
 (concat "\\`"
         (regexp-quote (expand-file-name "~/org/blog/"))
         "[^/]+\\.org\\'")
 :trigger (lambda () (my/insert-site-file-header "blog")))

(set-file-template!
 (concat "\\`"
         (regexp-quote (expand-file-name "~/org/now/"))
         "[^/]+\\.org\\'")
 :trigger (lambda () (my/insert-site-file-header "now")))

(set-file-template!
 (concat "\\`"
         (regexp-quote (expand-file-name "~/org/time/"))
         ".+\\.org\\'")
 :trigger (lambda () (my/insert-site-file-header "time")))

;; Whenever you reconfigure a package, make sure to wrap your config in an
;; `after!' block, otherwise Doom's defaults may override your settings. E.g.
;;
;;   (after! PACKAGE
;;     (setq x y))
;;
;; The exceptions to this rule:
;;
;;   - Setting file/directory variables (like `org-directory')
;;   - Setting variables which explicitly tell you to set them before their
;;     package is loaded (see 'C-h v VARIABLE' to look up their documentation).
;;   - Setting doom variables (which start with 'doom-' or '+').
;;
;; Here are some additional functions/macros that will help you configure Doom.
;;
;; - `load!' for loading external *.el files relative to this one
;; - `use-package!' for configuring packages
;; - `after!' for running code after a package has loaded
;; - `add-load-path!' for adding directories to the `load-path', relative to
;;   this file. Emacs searches the `load-path' when you load packages with
;;   `require' or `use-package'.
;; - `map!' for binding new keys
;;
;; To get information about any of these functions/macros, move the cursor over
;; the highlighted symbol at press 'K' (non-evil users must press 'C-c c k').
;; This will open documentation for it, including demos of how they are used.
;; Alternatively, use `C-h o' to look up a symbol (functions, variables, faces,
;; etc).
;;
;; You can also try 'gd' (or 'C-c c d') to jump to their definition and see how
;; they are implemented.

;;; magit
(after! magit
  (setq magit-diff-refine-hunk 'all
        magit-diff-highlight-trailing nil
        magit-diff-paint-whitespace nil))

(custom-set-faces!
  '(magit-diff-added             :background unspecified)
  '(magit-diff-added-highlight   :background unspecified)
  '(magit-diff-removed           :background unspecified)
  '(magit-diff-removed-highlight :background unspecified))

;;; difftastic - AST-aware diffs, called on demand from magit
(use-package! difftastic
  :defer t
  :init
  (map! :leader
        :prefix "g"
        :desc "Difftastic show commit"   "D" #'difftastic-magit-show
        :desc "Difftastic diff range"    "T" #'difftastic-magit-diff)
  ;; Note: difftastic lands in doom's bottom popup (small). Press `C-w _` to
  ;; maximize the window vertically, or `C-w o` to fill the frame.
  )

Brewfile entry for the difft binary used by difftastic.el.

brew "difftastic"
;;; org-glossary-export-fix.el --- Preserve only authored glossary breaks -*- lexical-binding: t; -*-

(require 'org-glossary)

(defun org-glossary-export-fix--trim-paragraph (value _backend _info)
  "Remove a single structural ending newline from inline glossary VALUE.
Work on a copied paragraph, preserving authored internal and explicit
line breaks.  Multi-paragraph and other block definitions are unchanged."
  (when (and (consp value)
             (null (cdr value))
             (eq (org-element-type (car value)) 'paragraph))
    (let* ((paragraph (org-element-copy (car value) t))
           (last-child (last (org-element-contents paragraph))))
      (when (and (stringp (car last-child))
                 (string-suffix-p "\n" (car last-child)))
        (setcar last-child (substring (car last-child) 0 -1))
        (list paragraph)))))

(defun org-glossary-export-fix--export-term (original entry info)
  "Export glossary ENTRY with ORIGINAL using an isolated copy of INFO.
Normalize only inline glossary paragraphs with preserved line breaks,
after existing parse-tree filters.  Keep the original export cache."
  (if (not (plist-get info :preserve-breaks))
      (funcall original entry info)
    (let ((export-info (copy-sequence info)))
      (setq export-info
            (plist-put
             export-info :filter-parse-tree
             (append
              (remq #'org-glossary-export-fix--trim-paragraph
                    (plist-get info :filter-parse-tree))
              (list #'org-glossary-export-fix--trim-paragraph))))
      (funcall original entry export-info))))

;; org-glossary exports a single paragraph's children inline.  With \n:t,
;; its structural ending newline otherwise becomes an HTML/LaTeX break.
(unless (advice-member-p #'org-glossary-export-fix--export-term
                        'org-glossary--export-term)
  (advice-add 'org-glossary--export-term :around
              #'org-glossary-export-fix--export-term))

(provide 'org-glossary-export-fix)
;;; org-glossary-export-fix.el ends here
init.el
;;; init.el -*- lexical-binding: t; -*-

;; This file controls what Doom modules are enabled and what order they load
;; in. Remember to run 'doom sync' after modifying it!

;; NOTE Press 'SPC h d h' (or 'C-h d h' for non-vim users) to access Doom's
;;      documentation. There you'll find a link to Doom's Module Index where all
;;      of our modules are listed, including what flags they support.

;; NOTE Move your cursor over a module's name (or its flags) and press 'K' (or
;;      'C-c c k' for non-vim users) to view its documentation. This works on
;;      flags as well (those symbols that start with a plus).
;;
;;      Alternatively, press 'gd' (or 'C-c c d') on a module to browse its
;;      directory (for easy access to its source code).

(doom! :input
       ;;(colemak +mod-dh)
       ;;bidi              ; (tfel ot) thgir etirw uoy gnipleh
       ;;chinese
       ;;japanese
       ;;layout            ; auie,ctsrnm is the superior home row

       :completion
       ;;company           ; the ultimate code completion backend
       ;;(corfu +orderless)  ; complete with cap(f), cape and a flying feather!
       ;;helm              ; the *other* search engine for love and life
       ;;ido               ; the other *other* search engine...
       ;;ivy               ; a search engine for love and life
       vertico           ; the search engine of the future

       :ui
       ;;deft              ; notational velocity for Emacs
       doom              ; what makes DOOM look the way it does
       doom-dashboard    ; a nifty splash screen for Emacs
       doom-quit         ; DOOM quit-message prompts when you quit Emacs
       ;;(emoji +unicode)  ; 🙂
       hl-todo           ; highlight TODO/FIXME/NOTE/DEPRECATED/HACK/REVIEW
       indent-guides     ; highlighted indent columns
       ligatures         ; ligatures and symbols to make your code pretty again
       ;;minimap           ; show a map of the code on the side
       modeline          ; snazzy, Atom-inspired modeline, plus API
       nav-flash         ; blink cursor line after big motions
       ;;neotree           ; a project drawer, like NERDTree for vim
       ophints           ; highlight the region an operation acts on
       (popup +all +defaults)   ; tame sudden yet inevitable temporary windows
       ;;smooth-scroll     ; So smooth you won't believe it's not butter
       ;;tabs              ; a tab bar for Emacs
       treemacs          ; a project drawer, like neotree but cooler
       ;;unicode           ; extended unicode support for various languages
       (vc-gutter +pretty) ; vcs diff in the fringe
       vi-tilde-fringe   ; fringe tildes to mark beyond EOB
       window-select     ; visually switch windows
       workspaces        ; tab emulation, persistence & separate workspaces
       ;;zen               ; distraction-free coding or writing

       :editor
       (evil +everywhere); come to the dark side, we have cookies
       file-templates    ; auto-snippets for empty files
       fold              ; (nigh) universal code folding
       ;;(format +onsave)  ; automated prettiness
       ;;god               ; run Emacs commands without modifier keys
       ;;lispy             ; vim for lisp, for people who don't like vim
       ;;multiple-cursors  ; editing in many places at once
       ;;objed             ; text object editing for the innocent
       ;;parinfer          ; turn lisp into python, sort of
       ;;rotate-text       ; cycle region at point between text candidates
       snippets          ; my elves. They type so I don't have to
       ;;word-wrap         ; soft wrapping with language-aware indent

       :emacs
       dired             ; making dired pretty [functional]
       electric          ; smarter, keyword-based electric-indent
       ;;eww               ; the internet is gross
       ;;ibuffer           ; interactive buffer management
       undo              ; persistent, smarter undo for your inevitable mistakes
       vc                ; version-control and Emacs, sitting in a tree

       :term
       eshell            ; the elisp shell that works everywhere
       ;;shell             ; simple shell REPL for Emacs
       ;;term              ; basic terminal emulator for Emacs
       vterm             ; the best terminal emulation in Emacs

       :checkers
       syntax              ; tasing you for every semicolon you forget
       ;;(spell +flyspell) ; tasing you for misspelling mispelling
       ;;grammar           ; tasing grammar mistake every you make

       :tools
       ;;ansible
       biblio            ; Writes a PhD for you (citation needed)
       ;;collab            ; buffers with friends
       ;;debugger          ; FIXME stepping through code, to help you add bugs
       ;;direnv
       ;;docker
       ;;editorconfig      ; let someone else argue about tabs vs spaces
       ;;ein               ; tame Jupyter notebooks with emacs
       (eval +overlay)     ; run code, run (also, repls)
       lookup              ; navigate your code and its documentation
       ;;llm               ; when I said you needed friends, I didn't mean...
       lsp               ; M-x vscode
       magit             ; a git porcelain for Emacs
       make              ; run make tasks from Emacs
       ;;pass              ; password manager for nerds
       (pdf +noter)      ; pdf enhancements with org-noter annotations
       ;;terraform         ; infrastructure as code
       tmux              ; an API for interacting with tmux
       tree-sitter       ; syntax and parsing, sitting in a tree...
       ;;upload            ; map local to remote projects via ssh/ftp

       :os
       (:if (featurep :system 'macos) macos)  ; improve compatibility with macOS
       ;;tty               ; improve the terminal Emacs experience

       :lang
       ;;ada               ; In strong typing we (blindly) trust
       ;;agda              ; types of types of types of types...
       ;;beancount         ; mind the GAAP
       (cc +lsp)         ; C > C++ == 1
       ;;clojure           ; java with a lisp
       ;;common-lisp       ; if you've seen one lisp, you've seen them all
       ;;coq               ; proofs-as-programs
       ;;crystal           ; ruby at the speed of c
       ;;csharp            ; unity, .NET, and mono shenanigans
       data              ; config/data formats
       ;;(dart +flutter)   ; paint ui and not much else
       ;;dhall
       elixir            ; erlang done right
       ;;elm               ; care for a cup of TEA?
       emacs-lisp        ; drown in parentheses
       ;;erlang            ; an elegant language for a more civilized age
       ;;ess               ; emacs speaks statistics
       ;;factor
       ;;faust             ; dsp, but you get to keep your soul
       ;;fortran           ; in FORTRAN, GOD is REAL (unless declared INTEGER)
       ;;fsharp            ; ML stands for Microsoft's Language
       ;;fstar             ; (dependent) types and (monadic) effects and Z3
       ;;gdscript          ; the language you waited for
       ;;(go +lsp)         ; the hipster dialect
       ;;(graphql +lsp)    ; Give queries a REST
       (haskell +lsp)    ; a language that's lazier than I am
       ;;hy                ; readability of scheme w/ speed of python
       ;;idris             ; a language you can depend on
       json              ; At least it ain't XML
       ;;janet             ; Fun fact: Janet is me!
       ;;(java +lsp)       ; the poster child for carpal tunnel syndrome
       javascript        ; all(hope(abandon(ye(who(enter(here))))))
       ;;julia             ; a better, faster MATLAB
       ;;kotlin            ; a better, slicker Java(Script)
       latex             ; writing papers in Emacs has never been so fun
       ;;lean              ; for folks with too much to prove
       ;;ledger            ; be audit you can be
       ;;lua               ; one-based indices? one-based indices
       markdown          ; writing docs for people to ignore
       ;;nim               ; python + lisp at the speed of c
       ;;nix               ; I hereby declare "nix geht mehr!"
       ;;ocaml             ; an objective camel
       (org               ; organize your plain life in plain text
         +attach
         +babel
         +capture
         +export
         +roam
         ;; +present
       )
       ;; php               ; perl's insecure younger brother
       ;;plantuml          ; diagrams for confusing people more
       ;;graphviz          ; diagrams for confusing yourself even more
       ;;purescript        ; javascript, but functional
       python            ; beautiful is better than ugly
       ;;qt                ; the 'cutest' gui framework ever
       ;;racket            ; a DSL for DSLs
       ;;raku              ; the artist formerly known as perl6
       ;;rest              ; Emacs as a REST client
       ;;rst               ; ReST in peace
       (ruby +rails)     ; 1.step {|i| p "Ruby is #{i.even? ? 'love' : 'life'}"}
       (rust +lsp)       ; Fe2O3.unwrap().unwrap().unwrap().unwrap()
       ;;scala             ; java, but good
       ;;(scheme +guile)   ; a fully conniving family of lisps
       sh                ; she sells {ba,z,fi}sh shells on the C xor
       ;;sml
       ;;solidity          ; do you need a blockchain? No.
       swift             ; who asked for emoji variables?
       ;;terra             ; Earth and Moon in alignment for performance.
       web               ; the tubes
       yaml              ; JSON, but readable
       ;;zig               ; C, but simpler

       :email
       ;;(mu4e +org +gmail)
       ;;notmuch
       ;;(wanderlust +gmail)

       :app
       ;;calendar
       ;;emms
       ;;everywhere        ; *leave* Emacs!? You must be joking
       ;;irc               ; how neckbeards socialize
       ;;(rss +org)        ; emacs as an RSS reader

       :config
       ;;literate
       (default +bindings +smartparens)) ; TODO +snippets +evil-commands
packages.el
;; -*- no-byte-compile: t; -*-
;;; $DOOMDIR/packages.el

;; To install a package with Doom you must declare them here and run 'doom sync'
;; on the command line, then restart Emacs for the changes to take effect -- or
;; use 'M-x doom/reload'.

;; To install SOME-PACKAGE from MELPA, ELPA or emacsmirror:
;; (package! some-package)

;; org
(package! org-appear)
(package! org-transclusion)
(package! org-contacts
  :recipe (:host github :repo "emacsmirror/org-contacts")
  :pin "7b93522b6f9d0f3343fd34e5fd1be2ce2f66b279")

;; citations and bibliography
(package! citar)
(package! citar-org-roam)

;; pdf annotation
(package! org-noter)

;; roam graph visualization
(package! org-roam-ui)

;; glossary and acronym management
(package! org-glossary
  :recipe (:host github :repo "tecosaur/org-glossary"))

;; structural (AST-aware) diffs on demand from magit
(package! difftastic)


;; To install a package directly from a remote git repo, you must specify a
;; `:recipe'. You'll find documentation on what `:recipe' accepts here:
;; https://github.com/radian-software/straight.el#the-recipe-format
;; (package! another-package
;;   :recipe (:host github :repo "username/repo"))

;; If the package you are trying to install does not contain a PACKAGENAME.el
;; file, or is located in a subdirectory of the repo, you'll need to specify
;; `:files' in the `:recipe':
;; (package! this-package
;;   :recipe (:host github :repo "username/repo"
;;            :files ("some-file.el" "src/lisp/*.el")))

;; If you'd like to disable a package included with Doom, you can do so here
;; with the `:disable' property:
;; (package! builtin-package :disable t)

;; You can override the recipe of a built in package without having to specify
;; all the properties for `:recipe'. These will inherit the rest of its recipe
;; from Doom or MELPA/ELPA/Emacsmirror:
;; (package! builtin-package :recipe (:nonrecursive t))
;; (package! builtin-package-2 :recipe (:repo "myfork/package"))

;; Specify a `:branch' to install a package from a particular branch or tag.
;; This is required for some packages whose default branch isn't 'master' (which
;; our package manager can't deal with; see radian-software/straight.el#279)
;; (package! builtin-package :recipe (:branch "develop"))

;; Use `:pin' to specify a particular commit to install.
;; (package! builtin-package :pin "1a2b3c4d5e")


;; Doom's packages are pinned to a specific commit and updated from release to
;; release. The `unpin!' macro allows you to unpin single packages...
;; (unpin! pinned-package)
;; ...or multiple packages
;; (unpin! pinned-package another-pinned-package)
;; ...Or *all* packages (NOT RECOMMENDED; will likely break things)
;; (unpin! t)
Snippets
# name: Full name: Anton Sebastian Bilbaeno
# key: asb
# --
Anton Sebastian Bilbaeno

Tealdeer

Community-maintained tldr pages. Quick command reference.

brew "tealdeer"
# Tealdeer
export TEALDEER_CONFIG_DIR=$HOME/.config/tealdeer
[style.description]
underline = false
bold = false
italic = false

[style.command_name]
foreground = "cyan"
underline = false
bold = false
italic = false

[style.example_text]
foreground = "green"
underline = false
bold = false
italic = false

[style.example_code]
foreground = "cyan"
underline = false
bold = false
italic = false

[style.example_variable]
foreground = "cyan"
underline = true
bold = false
italic = false

[display]
compact = false
use_pager = false

[updates]
auto_update = true
auto_update_interval_hours = 720

[directories]

mise

Version manager for dev tools

brew "mise"
# mise
eval "$(mise activate zsh)"

Haskell

# haskell
export PATH="$HOME/.ghcup/bin:$PATH"
export PATH="$HOME/.local/bin:$PATH"

JankyBorders

Window borders for the tiling window manager.

tap "FelixKratz/formulae", trusted: { formula: "borders" }
brew "FelixKratz/formulae/borders"

Zed

Installation

cask "zed"

Settings

// Zed settings
//
// For information on how to configure Zed, see the Zed
// documentation: https://zed.dev/docs/configuring-zed
//
// To see all of Zed's default settings without changing your
// custom settings, run `zed: open default settings` from the
// command palette (cmd-shift-p / ctrl-shift-p)
{
  "agent_servers": {
    "codex-acp": {
      "default_config_options": {
        "reasoning_effort": "low"
      },
      "type": "registry"
    }
  },
  "indent_guides": {
    "line_width": 1
  },
  "edit_predictions": {
    "allow_data_collection": "default",
    "mode": "subtle",
    "provider": "zed"
  },
  "context_servers": {
    "mcp-server-github": {
      "enabled": false,
      "remote": false,
      "settings": {}
    },
    "kagimcp": {
      "enabled": false,
      "remote": false,
      "settings": {}
    },
    "mcp-server-context7": {
      "enabled": false,
      "settings": {}
    }
  },
  "soft_wrap": "editor_width",
  "agent": {
    "sandbox_permissions": {
      "allow_fs_write_all": true
    },
    "single_file_review": true,
    "show_turn_stats": true,
    "tool_permissions": {
      "default": "allow"
    },
    "play_sound_when_agent_done": "never",
    "default_profile": "write",
    "default_model": {
      "effort": "low",
      "enable_thinking": true,
      "provider": "openai-subscribed",
      "model": "gpt-5.6-sol"
    },
    "model_parameters": []
  },
  "ui_font_size": 18.0,
  "buffer_font_size": 15.0,
  "theme": {
    "mode": "system",
    "light": "One Light",
    "dark": "One Dark"
  },
  "vim_mode": true,
  "relative_line_numbers": "enabled"
}

Keymap

// Zed keymap
//
// For information on binding keys, see the Zed
// documentation: https://zed.dev/docs/key-bindings
//
// To see the default key bindings run `zed: open default keymap`
// from the command palette.
[
  {
    "context": "Workspace",
    "bindings": {
      // "shift shift": "file_finder::Toggle"
    }
  },
  {
    "context": "Editor && vim_mode == insert",
    "bindings": {
      "j k": "vim::NormalBefore"
    }
  }
  // {
  //   "context": "Editor && vim_mode == normal && vim_operator == none",
  //   "bindings": {
  //     "n": "vim::Left",
  //     "e": "vim::Down",
  //     "i": "vim::Up",
  //     "o": "vim::Right"
  //   }
  // }
]

Ghostty

Terminal emulator.

cask "ghostty"
# This is the configuration file for Ghostty.
#
# This template file has been automatically created at the following
# path since Ghostty couldn't find any existing config files on your system:
#
#   /Users/antoaenono/Library/Application Support/com.mitchellh.ghostty/config
#
# The template does not set any default options, since Ghostty ships
# with sensible defaults for all options. Users should only need to set
# options that they want to change from the default.
#
# Run `ghostty +show-config --default --docs` to view a list of
# all available config options and their default values.
#
# Additionally, each config option is also explained in detail
# on Ghostty's website, at https://ghostty.org/docs/config.

# Config syntax crash course
# ==========================
# # The config file consists of simple key-value pairs,
# # separated by equals signs.
# font-family = Iosevka
# window-padding-x = 2
#
# # Spacing around the equals sign does not matter.
# # All of these are identical:
# key=value
# key= value
# key =value
# key = value
#
# # Any line beginning with a # is a comment. It's not possible to put
# # a comment after a config option, since it would be interpreted as a
# # part of the value. For example, this will have a value of "#123abc":
# background = #123abc
#
# # Empty values are used to reset config keys to default.
# key =
#
# # Some config options have unique syntaxes for their value,
# # which is explained in the docs for that config option.
# # Just for example:
# resize-overlay-duration = 4s 200ms

font-size=28

theme = Monokai Pro Machine

window-save-state = always
macos-titlebar-style = transparent
copy-on-select = clipboard

keybind = cmd+s>r=reload_config

Utilities

cask "keycastr"
brew "gnupg"
brew "pinentry-mac"

# Local screen/webcam recording and scripted video processing.
cask "obs"
brew "ffmpeg"

Espanso

cask "espanso"

On first install, espanso defaults to ~/Library/Application Support/espanso.
Move it so chezmoi can manage it via ~/.config/espanso:

espanso stop
mv "$HOME/Library/Application Support/espanso" "$HOME/.config/espanso"

Config and snippets are tangled from Cross-platform > Espanso.

Cross-platform

Espanso

Cross-platform text expander.

Config
# Espanso default configuration
# https://espanso.org/docs/configuration/basics/
backend: auto
Matches
# trigger: abbreviation you type
# replace: expanded text
matches:
  - trigger: ";hello"
    replace: "Hello, how are you?"

  - trigger: "--dang"
    replace: "--dangerously-skip-permissions"

Agents

Cross-agent instructions, skills, and user-owned hooks live under ~/.agents/.
Product-specific directories contain only adapters and configuration that cannot
be shared.

Repository Instructions
# Dotfiles Repository Guidance

## Source of Truth

- `config.org` is a literate configuration file and the authoritative source for configuration and documentation.
- Do not edit tangled files directly. Instead, configuration code should be tangled from an Org code block.
- Keep generated paths and their `:tangle` targets aligned.
- Prose outside Org source blocks is human-authored and must not be edited by agents.

## Change workflow

1. Write configuration code only in Org source blocks in `config.org`. Ask the user to make any needed prose or heading changes.
2. Follow the globally installed `$configure-dotfiles` skill to tangle, review, and apply the change.
3. Review the generated diff, including file modes and templates.
4. Run the relevant validation or dry-run before applying with `chezmoi apply`.

## Safety

- Preserve existing user changes and inspect unexpected diffs before overwriting anything.
- Never place secrets, tokens, private keys, or machine-specific credentials in this repository.
- Prefer small, reversible changes and document non-obvious platform assumptions.

## Agent skill

For dotfiles changes, follow `dot_agents/skills/configure-dotfiles/SKILL.md`.
Global Instructions

The canonical global instructions merge the existing personal guidance used by
different coding agents.

You are capable of excellent work; favor results that are correct, simple, coherent, and well validated.

## Resolving Ambiguity & Clarifying Information

- Ask follow-up questions before implementation when ambiguity affects correctness, scope, domain language, public contracts, data loss, irreversible changes, or durable architectural decisions.
- For low-risk ambiguity, make a reasonable assumption, state it briefly, and proceed.
- If the requested outcome conflicts with existing domain language or architecture, clarify the intended direction before changing code.

## Anti-Sycophancy & Epistemic Stability

- Be helpful without being agreeable by default. Do not validate a user's claim, premise, plan, or interpretation merely because they stated it confidently or repeatedly.
- Reason independently from the user's stated preference. Treat confidence, repetition, social proof, authority, and urgency as signals to re-check evidence, not as reasons to agree.
- Maintain a justified technical stance across turns unless new evidence, code, tests, documentation, or explicit requirements warrant changing it. If you change your view, state what changed and why.
- When pressure or disagreement could be affecting the answer, briefly adopt a neutral reviewer stance: assess the claim as if reviewing a third party's technical argument, then answer in your normal assistant voice.
- Detect false presuppositions and hidden assumptions in requests. Correct them directly before building on the request, especially when they affect correctness, safety, data loss, public contracts, or architecture.
- For ethically risky, biased, unsafe, or discriminatory premises, identify the problematic assumption and redirect to a safer, accurate formulation instead of complying with the premise.
- When the user challenges your answer, re-check the evidence and reasoning. If the original answer remains sound, say so plainly and offer the strongest actionable path forward.
- Separate tone from truth: stay respectful, concise, and collaborative while preserving factual accuracy, engineering rigor, and safety.

# Style

Never use em dashes (—). Prefer commas, colons, semicolons, or regular dashes (-) instead.

# Git

## Branching

Prefer working on a branch rather than committing directly to main. Exception: personal/dotfiles repos where committing to main is acceptable.

## Committing

Review status and diffs before committing. Create small, coherent commits and
stage explicit paths. When changes clearly belong to existing unpushed commits,
consider `git absorb` and inspect the autosquashed history. Never rewrite pushed
history or force-push without explicit permission.

## Commits

Never include "Co-Authored-By" lines in commits.
Never include agent-generation attribution in commits or pull request descriptions.

## Pull request body template

When creating or editing a PR, look for a `PULL_REQUEST_TEMPLATE.md` in `.github/` or the repo root and use it as the body structure. Fill in each section with real content. Reproduce all checklist items, checking off satisfied ones (`[x]`) and leaving the rest unchecked (`[ ]`). Never remove unchecked items or add sections not in the template.
{{ .chezmoi.homeDir }}/.agents/AGENTS.md
{{ .chezmoi.homeDir }}/.agents/AGENTS.md
Hooks

User-owned cross-agent hooks live in ~/.agents/hooks/. Product configuration
must point to scripts there. Externally managed hooks remain product-local so
their installers can update them safely.

The literate configuration reflector snapshots config.org before every Codex
tool call. After the tool completes, it compares the ordered bytes outside Org
source blocks. If those human-authored bytes changed, it restores the complete
file and its previous Git index entry. Repository-local registration keeps this
policy separate from global and externally managed hooks.

{
  "description": "Protect the human-authored prose in the literate dotfiles configuration.",
  "hooks": {
    "PreToolUse": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "/usr/bin/python3 \"$HOME/.agents/hooks/protect_literate_config.py\"",
            "timeout": 10,
            "statusMessage": "Snapshotting literate configuration prose"
          }
        ]
      }
    ],
    "PostToolUse": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "/usr/bin/python3 \"$HOME/.agents/hooks/protect_literate_config.py\"",
            "timeout": 10,
            "statusMessage": "Checking literate configuration prose"
          }
        ]
      }
    ]
  }
}
"""Restore config.org when a Codex tool call changes human-authored prose."""

from __future__ import annotations

import hashlib
import json
import os
import re
import shutil
import stat
import subprocess
import sys
import tempfile
import time
import unittest
from pathlib import Path
from typing import Any

TARGET_PATH = "config.org"
SNAPSHOT_VERSION = 1
STALE_AFTER_SECONDS = 24 * 60 * 60
BEGIN_SOURCE = re.compile(br"[ \t]*#\+begin_src(?:[ \t].*)?", re.IGNORECASE)
END_SOURCE = re.compile(br"[ \t]*#\+end_src[ \t]*", re.IGNORECASE)


class GuardError(RuntimeError):
    """The protected state could not be snapshotted or restored safely."""


def main() -> int:
    if sys.argv[1:] == ["--self-test"]:
        suite = unittest.defaultTestLoader.loadTestsFromTestCase(GuardTests)
        return 0 if unittest.TextTestRunner(verbosity=2).run(suite).wasSuccessful() else 1

    try:
        payload = json.load(sys.stdin)
        cwd = Path(str(payload.get("cwd") or ".")).resolve()
        repo = find_repository(cwd)
        if repo is None:
            return 0

        tool_use_id = str(payload.get("tool_use_id") or "")
        if not tool_use_id:
            raise GuardError("hook input has no tool_use_id")

        root = snapshot_root(repo)
        root.mkdir(parents=True, exist_ok=True, mode=0o700)
        cleanup_stale_snapshots(root)
        snapshot = root / safe_id(tool_use_id)
        event = payload.get("hook_event_name")

        if event == "PreToolUse":
            create_snapshot(repo, snapshot, payload)
            return 0
        if event == "PostToolUse":
            message = validate_and_restore(repo, snapshot, tool_use_id)
            if message:
                print(message, file=sys.stderr)
                return 2
            return 0
        raise GuardError(f"unsupported hook event: {event!r}")
    except (GuardError, json.JSONDecodeError, OSError, subprocess.CalledProcessError) as error:
        print(f"Literate config reflector failed: {error}", file=sys.stderr)
        return 2


def find_repository(cwd: Path) -> Path | None:
    try:
        result = run_git(cwd, "rev-parse", "--show-toplevel")
    except subprocess.CalledProcessError:
        return None
    return Path(result.stdout.decode().strip()).resolve()


def run_git(
    repo: Path, *args: str, input: bytes | None = None
) -> subprocess.CompletedProcess[bytes]:
    return subprocess.run(
        ["git", "-C", str(repo), *args],
        input=input,
        check=True,
        capture_output=True,
    )


def snapshot_root(repo: Path) -> Path:
    result = run_git(repo, "rev-parse", "--git-path", "codex-hooks/protect-literate-config")
    path = Path(result.stdout.decode().strip())
    return path if path.is_absolute() else (repo / path).resolve()


def safe_id(value: str) -> str:
    return hashlib.sha256(value.encode()).hexdigest()


def protected_projection(content: bytes) -> bytes:
    """Return exact ordered bytes outside complete Org source blocks."""

    protected = bytearray()
    inside_source = False
    source_started_at = 0

    for line_number, line in enumerate(content.splitlines(keepends=True), 1):
        logical_line = line.rstrip(b"\r\n")
        if inside_source:
            if END_SOURCE.fullmatch(logical_line):
                inside_source = False
            continue
        if END_SOURCE.fullmatch(logical_line):
            raise GuardError(f"unmatched #+end_src at line {line_number}")
        if BEGIN_SOURCE.fullmatch(logical_line):
            inside_source = True
            source_started_at = line_number
            continue
        protected.extend(line)

    if inside_source:
        raise GuardError(f"unclosed #+begin_src at line {source_started_at}")
    return bytes(protected)


def cleanup_stale_snapshots(root: Path) -> None:
    cutoff = time.time() - STALE_AFTER_SECONDS
    for child in root.iterdir():
        try:
            if child.is_dir() and child.stat().st_mtime < cutoff:
                shutil.rmtree(child)
        except OSError:
            pass


def create_snapshot(repo: Path, destination: Path, payload: dict[str, Any]) -> None:
    target = repo / TARGET_PATH
    info = target.lstat()
    if not stat.S_ISREG(info.st_mode):
        raise GuardError(f"{TARGET_PATH} is not a regular file")

    content = target.read_bytes()
    projection = protected_projection(content)
    if destination.exists():
        shutil.rmtree(destination)
    temporary = destination.with_name(f".{destination.name}.{os.getpid()}.tmp")
    if temporary.exists():
        shutil.rmtree(temporary)
    temporary.mkdir(parents=True, mode=0o700)

    metadata = {
        "version": SNAPSHOT_VERSION,
        "created_at": time.time(),
        "repo": str(repo),
        "tool_use_id": payload.get("tool_use_id"),
        "session_id": payload.get("session_id"),
        "turn_id": payload.get("turn_id"),
        "mode": stat.S_IMODE(info.st_mode),
        "content_sha256": digest(content),
        "projection_sha256": digest(projection),
    }
    write_private(temporary / "config.org", content)
    write_private(
        temporary / "manifest.json",
        json.dumps(metadata, sort_keys=True, separators=(",", ":")).encode(),
    )
    write_private(temporary / "index.entries", protected_index(repo))
    write_private(temporary / "integrity.sha256", snapshot_integrity(temporary).encode())
    os.replace(temporary, destination)


def validate_and_restore(repo: Path, snapshot: Path, tool_use_id: str) -> str | None:
    if not snapshot.is_dir():
        raise GuardError("matching pre-tool snapshot is missing")

    recorded_integrity = (snapshot / "integrity.sha256").read_text()
    if snapshot_integrity(snapshot) != recorded_integrity:
        raise GuardError("snapshot content or metadata was modified")

    metadata = json.loads((snapshot / "manifest.json").read_text())
    if (
        metadata.get("version") != SNAPSHOT_VERSION
        or metadata.get("repo") != str(repo)
        or metadata.get("tool_use_id") != tool_use_id
    ):
        raise GuardError("snapshot metadata does not match this tool call")

    before = (snapshot / "config.org").read_bytes()
    if digest(before) != metadata.get("content_sha256"):
        raise GuardError("snapshotted config.org content was modified")
    before_projection = protected_projection(before)
    if digest(before_projection) != metadata.get("projection_sha256"):
        raise GuardError("snapshotted prose projection was modified")

    target = repo / TARGET_PATH
    after_projection: bytes | None = None
    try:
        if target.is_file() and not target.is_symlink():
            after_projection = protected_projection(target.read_bytes())
    except GuardError:
        after_projection = None

    if after_projection == before_projection:
        shutil.rmtree(snapshot)
        return None

    restore_file(target, before, int(metadata["mode"]))
    restore_index(repo, (snapshot / "index.entries").read_bytes())
    shutil.rmtree(snapshot)
    return (
        "Restored config.org because this model tool call changed human-authored "
        "content outside Org source blocks. Retry with a source-block-only edit."
    )


def digest(content: bytes) -> str:
    return hashlib.sha256(content).hexdigest()


def write_private(path: Path, content: bytes) -> None:
    descriptor = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600)
    with os.fdopen(descriptor, "wb") as stream:
        stream.write(content)


def snapshot_integrity(snapshot: Path) -> str:
    checksum = hashlib.sha256()
    for name in ("config.org", "manifest.json", "index.entries"):
        checksum.update(name.encode())
        checksum.update(b"\0")
        checksum.update((snapshot / name).read_bytes())
        checksum.update(b"\0")
    return checksum.hexdigest()


def protected_index(repo: Path) -> bytes:
    return run_git(repo, "ls-files", "--stage", "-z", "--", TARGET_PATH).stdout


def restore_file(target: Path, content: bytes, mode: int) -> None:
    target.parent.mkdir(parents=True, exist_ok=True)
    descriptor, temporary_name = tempfile.mkstemp(prefix=f".{target.name}.", dir=target.parent)
    temporary = Path(temporary_name)
    try:
        with os.fdopen(descriptor, "wb") as stream:
            stream.write(content)
            stream.flush()
            os.fsync(stream.fileno())
        os.chmod(temporary, mode)
        os.replace(temporary, target)
    finally:
        if temporary.exists():
            temporary.unlink()


def restore_index(repo: Path, entries: bytes) -> None:
    run_git(repo, "update-index", "--force-remove", "--", TARGET_PATH)
    if entries:
        run_git(repo, "update-index", "-z", "--index-info", input=entries)


class GuardTests(unittest.TestCase):
    def setUp(self) -> None:
        self.temporary = tempfile.TemporaryDirectory()
        self.repo = Path(self.temporary.name)
        run_git(self.repo, "init", "--quiet")
        run_git(self.repo, "config", "user.email", "lit@example.invalid")
        run_git(self.repo, "config", "user.name", "Lit Test")
        self.original = (
            b"* Human heading\nHuman prose.\n"
            b"#+begin_src shell :tangle ./example\n"
            b"echo before\n"
            b"#+end_src\n"
            b"Closing prose.\n"
        )
        (self.repo / TARGET_PATH).write_bytes(self.original)
        run_git(self.repo, "add", TARGET_PATH)
        run_git(self.repo, "commit", "--quiet", "-m", "initial")

    def tearDown(self) -> None:
        self.temporary.cleanup()

    def snapshot(self, tool_use_id: str = "tool-1") -> Path:
        destination = snapshot_root(self.repo) / safe_id(tool_use_id)
        destination.parent.mkdir(parents=True, exist_ok=True)
        create_snapshot(
            self.repo,
            destination,
            {"tool_use_id": tool_use_id, "session_id": "session", "turn_id": "turn"},
        )
        return destination

    def run_hook(self, event: str, tool_use_id: str = "tool-1") -> subprocess.CompletedProcess[bytes]:
        payload = {
            "cwd": str(self.repo),
            "hook_event_name": event,
            "tool_use_id": tool_use_id,
            "session_id": "session",
            "turn_id": "turn",
        }
        return subprocess.run(
            [sys.executable, str(Path(__file__).resolve())],
            input=json.dumps(payload).encode(),
            capture_output=True,
        )

    def test_projection_ignores_source_blocks(self) -> None:
        changed = self.original.replace(b"echo before", b"echo after\necho another")
        self.assertEqual(protected_projection(self.original), protected_projection(changed))

    def test_projection_detects_prose_changes(self) -> None:
        changed = self.original.replace(b"Human prose.", b"Model prose.")
        self.assertNotEqual(protected_projection(self.original), protected_projection(changed))

    def test_projection_rejects_malformed_source_blocks(self) -> None:
        with self.assertRaisesRegex(GuardError, "unclosed"):
            protected_projection(b"Human prose.\n#+begin_src shell\necho nope\n")
        with self.assertRaisesRegex(GuardError, "unmatched"):
            protected_projection(b"Human prose.\n#+end_src\n")

    def test_source_only_edit_survives(self) -> None:
        snapshot = self.snapshot()
        changed = self.original.replace(b"echo before", b"echo after")
        (self.repo / TARGET_PATH).write_bytes(changed)

        self.assertIsNone(validate_and_restore(self.repo, snapshot, "tool-1"))
        self.assertEqual((self.repo / TARGET_PATH).read_bytes(), changed)

    def test_prose_edit_restores_file_and_index(self) -> None:
        snapshot = self.snapshot()
        changed = self.original.replace(b"Human prose.", b"Model prose.").replace(
            b"echo before", b"echo after"
        )
        (self.repo / TARGET_PATH).write_bytes(changed)
        run_git(self.repo, "add", TARGET_PATH)

        message = validate_and_restore(self.repo, snapshot, "tool-1")

        self.assertIn("Restored config.org", message or "")
        self.assertEqual((self.repo / TARGET_PATH).read_bytes(), self.original)
        self.assertEqual(run_git(self.repo, "diff", "--cached", "--", TARGET_PATH).stdout, b"")

    def test_malformed_edit_restores_file(self) -> None:
        snapshot = self.snapshot()
        (self.repo / TARGET_PATH).write_bytes(b"Human prose.\n#+begin_src shell\necho nope\n")

        self.assertIsNotNone(validate_and_restore(self.repo, snapshot, "tool-1"))
        self.assertEqual((self.repo / TARGET_PATH).read_bytes(), self.original)

    def test_staging_preexisting_human_prose_is_allowed(self) -> None:
        human = self.original.replace(b"Human prose.", b"Human revision.")
        (self.repo / TARGET_PATH).write_bytes(human)
        snapshot = self.snapshot()
        run_git(self.repo, "add", TARGET_PATH)

        self.assertIsNone(validate_and_restore(self.repo, snapshot, "tool-1"))
        self.assertEqual(run_git(self.repo, "show", f":{TARGET_PATH}").stdout, human)

    def test_deleted_file_is_restored(self) -> None:
        snapshot = self.snapshot()
        (self.repo / TARGET_PATH).unlink()
        run_git(self.repo, "update-index", "--force-remove", "--", TARGET_PATH)

        self.assertIsNotNone(validate_and_restore(self.repo, snapshot, "tool-1"))
        self.assertEqual((self.repo / TARGET_PATH).read_bytes(), self.original)
        self.assertEqual(run_git(self.repo, "show", f":{TARGET_PATH}").stdout, self.original)

    def test_hook_entrypoint_restores_deleted_file(self) -> None:
        self.assertEqual(self.run_hook("PreToolUse").returncode, 0)
        (self.repo / TARGET_PATH).unlink()

        result = self.run_hook("PostToolUse")

        self.assertEqual(result.returncode, 2)
        self.assertIn(b"Restored config.org", result.stderr)
        self.assertEqual((self.repo / TARGET_PATH).read_bytes(), self.original)


if __name__ == "__main__":
    raise SystemExit(main())
omp (oh-my-pi)

The omp coding agent, installed globally via bun add -g @oh-my-pi/pi-coding-agent.
Bun drops its global binaries in ~/.bun/bin, so that needs to be on PATH.

# omp (oh-my-pi) / bun global bins
export PATH="$HOME/.bun/bin:$PATH"
Skills

Cross-agent skills are tangled to ~/.agents/skills/. Product adapters point to
the canonical skill directories.

configure-dotfiles
---
name: configure-dotfiles
description: Safely maintain personal dotfiles in the chezmoi-managed, Org-babel-tangled source repository. Use from any conversation or working directory to inspect, add, change, tangle, preview, or apply machine configuration.
---

# Configure Dotfiles

## Model

Run `chezmoi source-path` to locate the source repository. Use that directory as
the working directory for the entire task.

`config.org` is the source of truth. Files such as `dot_config/**`, `Brewfile`, and
`.chezmoiscripts/**` are tangled outputs. Keep prose and code together in the Org
document. Never edit a tangled output directly.

## Workflow

1. Inspect `git status --short`, the relevant Org section, and its current tangle
   target. Preserve unrelated user changes.
2. Edit only Org source blocks in `config.org`, including the exact `:tangle`
   target. New blocks may be added under an existing human-authored section. Ask
   the user to make any required prose or heading changes. Add a shebang when the
   target must be executable.
3. Run `~/.agents/skills/configure-dotfiles/scripts/tangle.sh`. It uses a running
   Emacs server when available and otherwise launches batch Emacs. Do not
   reproduce the underlying Emacs invocation manually.
4. Run `git diff --check`, `git status --short`, and `git diff`. Confirm every
   generated change follows from the requested Org edits.
5. Preview deployment with `chezmoi apply --dry-run --verbose --force`. On normal
   command-line workflows, use `--force` only with `--dry-run`; it reveals target
   drift without overwriting it.
6. If implementation was requested and the preview contains only intended
   changes, apply the narrowest affected target or directory with `chezmoi apply
   --verbose <target>`. The interactive `M-x tangle-moi` helper may use `--force`
   for its real apply only after showing the full forced dry-run and receiving
   explicit confirmation. This avoids a second prompt that requires a TTY.
7. Verify installed targets and report source changes, applied targets, and
   anything intentionally left unapplied.

`M-x tangle-moi` is the interactive interface to the same tangle core installed
at `~/.config/doom/lisp/tangle-moi.el`.

## Stop conditions

Stop if the tangle script reports unsaved changes in an Emacs buffer visiting
`config.org`; preserve that buffer and ask the user to resolve it. Ask before
applying when the preview would overwrite independently modified home files,
delete targets, change permissions unexpectedly, execute package or setup
scripts, or include changes outside the request. Do not apply when the user asked
only for review or diagnosis.

## Conventions

- Use chezmoi naming conventions for destination paths (`dot_`, `private_`,
  `symlink_`, and `.tmpl`).
- Keep platform-specific scripts under `.chezmoiscripts/<os>/` and guard them
  appropriately.
- Keep secrets and host-specific credentials out of the repository.
- Prefer idempotent scripts with `set -e` and explicit paths.
- Flag nearby prose that needs a human update; do not edit it.

## Validation checklist

- The Org source parses and tangles successfully.
- The generated diff contains no unrelated changes.
- Templates and scripts remain executable where intended.
- The chezmoi dry-run reports only expected target changes.
set -eu

elisp_file="${XDG_CONFIG_HOME:-$HOME/.config}/doom/lisp/tangle-moi.el"

if [ ! -r "$elisp_file" ]; then
  echo "Tangle failed: shared Emacs library not found at $elisp_file." >&2
  exit 1
fi

if command -v emacsclient >/dev/null 2>&1 \
    && emacsclient --eval t >/dev/null 2>&1; then
  escaped_elisp_file=$(printf '%s' "$elisp_file" \
    | sed 's/\\/\\\\/g; s/"/\\"/g')
  exec emacsclient --eval \
    "(progn (load \"$escaped_elisp_file\" nil t) (tangle-moi-tangle))"
fi

if ! command -v emacs >/dev/null 2>&1; then
  echo "Tangle failed: Emacs is not installed or not on PATH." >&2
  exit 127
fi

exec emacs --batch \
  --load "$elisp_file" \
  --funcall tangle-moi-tangle-batch
interface:
  display_name: "Configure Dotfiles"
  short_description: "Safely edit, tangle, and apply personal dotfiles"
  default_prompt: "Use $configure-dotfiles to update my literate dotfiles and safely apply the result."
{{ .chezmoi.homeDir }}/.agents/skills/configure-dotfiles
tangle-moi
---
name: tangle-moi
description: Tangle the literate chezmoi config, preview the target diff, and optionally apply it. Use when explicitly invoking the agent equivalent of M-x tangle-moi.
---

# Tangle Moi

Use this skill as the explicit manual shortcut for the `M-x tangle-moi`
workflow.

Read `~/.agents/skills/configure-dotfiles/SKILL.md` and follow its workflow.
Run `~/.agents/skills/tangle-moi/tangle.sh` for the tangle step, preview the
chezmoi changes, and always ask before applying. Never auto-apply.
set -eu
exec "$HOME/.agents/skills/configure-dotfiles/scripts/tangle.sh"
interface:
  display_name: "Tangle Moi"
  short_description: "Tangle and preview the literate dotfiles config"
  default_prompt: "Use $tangle-moi to tangle my literate dotfiles and preview the chezmoi changes."
{{ .chezmoi.homeDir }}/.agents/skills/tangle-moi
citation-add
---
name: citation-add
description: Add new PDFs in ~/org/pdfs/ to references.bib. Cleans filenames, classifies, hits CrossRef for authoritative metadata, checks for duplicates, writes a validated bib entry, renames the PDF to the citekey, and runs biber. Zero verification debt - every entry is correct on arrival.
user-invocable: true
allowed-tools: Bash, Read, Edit
---

# citation-add

Adds every untracked PDF in `~/org/pdfs/` to `~/org/bib/references.bib`.
Process one PDF completely before moving to the next.
Entry quality is baked in at add time - no separate verify pass needed.

---

## Step 1: Clean and inventory

Strip Anna's Archive suffixes:

```bash
python3 ~/.agents/skills/citation-add/clean_filenames.py --apply
```

Find untracked PDFs:

```bash
python3 ~/.agents/skills/citation-add/find_missing.py
```

- `MISSING:<csv>` - filenames to process (stop here if empty)
- `CITEKEYS:<csv>` - existing citekeys (keep for collision checks)

---

## Step 2: For each missing PDF - one at a time

### 2a. Classify and extract

```bash
python3 ~/.agents/skills/citation-add/classify_pdf.py \
  --pdf ~/org/pdfs/<filename> \
  --task extract:bib
```

Parse output:
- `CLASS:<class>` - suggested document class; treat as a hint only. Use content
  and venue to determine the correct BibTeX entry type (`@article`, `@inproceedings`,
  `@incollection`, `@book`, `@phdthesis`, etc.) - CLASS can be wrong. If it is,
  fix the rule in `classify_pdf.py` and note the case in `classify.md`.
- `DOI:<doi>` - go to Step 3a if present
- `META:title=...` / `META:author=...` - PDF metadata; use it when it is a real
  value (it is sometimes a file name like `main-all.dvi`)
- `FIELDS:<csv>` - fields still needed from page content
- `RANGE:<range>` - pages to read for fallback
- `NOTEXT:...` - scanned/encrypted; do NOT skip. Instead:
  1. Check the filename for embedded metadata (Anna's Archive names include author, title, year, publisher, ISBN)
  2. Use world knowledge for well-known works (classic books, famous papers)
  3. Only escalate to the user if neither source yields enough to build a complete entry

### 2b. Duplicate check

Once DOI or title is known, run:

```bash
python3 ~/.agents/skills/citation-add/check_duplicate.py \
  --bib ~/org/bib/references.bib \
  --doi "<doi-or-empty>" \
  --title "<title-or-empty>" \
  --author "<author-or-empty>" \
  --year "<year-or-empty>" \
  --new-pdf "<filename>"
```

Interpret the output token:

- `NO_MATCH` - no duplicate found, proceed to Step 3
- `KEEP_EXISTING: citekey=... existing_file=... reason=...`
  - Delete the new PDF silently, log the decision in the Step 8 report
  - Do NOT add a new bib entry
- `KEEP_NEW: citekey=... existing_file=... reason=...`
  - Update the existing bib entry's `file` field to point to the new PDF
  - Delete the old PDF file, log the decision
  - Do NOT add a new bib entry
- `INFER: citekey=... existing_file=... new_pdf=... hint=...`
  - Read pages 1-2 of both PDFs using nav.py or Read
  - Reason about whether they are the same work, using the `hint` as guidance
  - Resolve as KEEP_EXISTING, KEEP_NEW, or escalate to ASK if still unclear
  - If resolved, log the decision; if escalating, show the user the evidence
- `ASK: citekey=... existing_file=... message=...`
  - Show the user the message and wait for their choice before proceeding

**Never silently overwrite a `file` field** without going through this flow.

---

## Step 3a: DOI path (preferred)

```bash
python3 ~/.agents/skills/citation-add/fetch_doi.py <doi>
```

Parse `FIELD:key=value` lines. Also watch for:
- `SPARSE:...` - CrossRef returned fewer than 4 substantive fields; read PDF pages
  to fill gaps before building the entry.

Then:

**CrossRef year is authoritative.** Use it for both the `year` field and the
citekey year. If CrossRef returns `published-print` and `published-online` in
different years, use `published-print`. If only `published-online` is present,
use that.

Check citekey year matches CrossRef year before writing. If they differ, use
CrossRef year in both the citekey and the `year` field.

Format following `~/.agents/skills/citation-add/examples.md`:
- `journal` for articles (not `journaltitle`)
- `publisher` in `note = {Publisher: X}` for `@article`
- `publisher = {IEEE}` / `publisher = {ACM}` valid for `@inproceedings`
- `abstract`: follow **Abstract** below

Add `file = {~/org/pdfs/<filename>}` (temporary - will be updated after rename).

---

## Step 3b: Fallback path (no DOI)

1. Read the `RANGE:` pages from the PDF
2. Use `META:` fields as starting point; fill `FIELDS:` from page content
3. Get the `abstract` as described in **Abstract** below, never by inference
4. Read `~/.agents/skills/citation-add/examples.md` for formatting conventions
5. Build entry manually

**Citekey convention:** `authoryear-slug`
- `year` = publication year (not submission/acceptance year)
- `slug` = 1-4 hyphenated words that answer "what would I search for?"
- Drop filler words: "towards", "combining", "using", "a", "an", "the", "on"
- Example: "Towards Responsible Software Engineering: combining value-based
  processes..." → `kapferer2024responsible-se`
- Check `CITEKEYS` for collisions; append `b`, `c` if needed

---

## Abstract (both paths)

The site shows `abstract` under an "Abstract" label as the paper's own text, so
it is either the work's own abstract, verbatim and complete, or omitted. Never
write it yourself: no paraphrase, summary, condensing, or truncation, and never
end it with "...". Summaries go in `annote` (Step 5).

Take the first source that has it:

1. CrossRef: the `FIELD:abstract=` line from `fetch_doi.py`.
2. arXiv, for `10.48550/arXiv.*` DOIs: the `<summary>` of
   `https://export.arxiv.org/api/query?id_list=<arxiv-id>`.
3. The PDF:

   ```bash
   python3 ~/.agents/skills/citation-add/extract_abstract.py ~/org/pdfs/<filename>
   ```

   - `ABSTRACT:<text>` - use it, after the check below
   - `NOABSTRACT:<reason>` - omit the field
   - `FAIL:<reason>` - get the pages to read from
     `classify_pdf.py --pdf <pdf> --task extract:abstract` (`patterns.md` notes
     where abstracts sit by document type) and copy the abstract exactly as
     printed. If a new pattern would have caught it, add the pattern to
     `extract_abstract.py` and record it in `patterns.md`.

Before writing it, check the text against the PDF page:

- It is the whole abstract. Nothing between it and the next heading
  (Keywords, Index Terms, CCS Concepts, 1 Introduction) is missing.
- No text-layer damage: dropped ligatures (`dene` for `define`), run-together
  or split words, lost line-end hyphens, OCR errors. Fix them to match the page.
- No publisher boilerplate: drop trailing copyright lines
  (`© 2015 Elsevier B.V. All rights reserved.`), received/accepted dates,
  funding or supplemental-material notes, and keyword lists.

Omit `abstract` when the work has none: books, theses whose only summary runs
over several pages, and columns, essays or magazine articles without one. For a
magazine article with no printed abstract, the publisher's registered abstract
(CrossRef, IEEE Xplore) may be used if it reads coherently; otherwise omit it.

---

## Step 4: Write entry and rename PDF

Append the completed entry to the bib.

Rename the PDF to `{citekey}.pdf`:

```python
# Always use Python for renames - shell mv fails on non-ASCII and special chars
import os
pdfs = os.path.expanduser('~/org/pdfs')
files = os.listdir(pdfs)
src = [f for f in files if f == '<original-filename>'][0]
os.rename(os.path.join(pdfs, src), os.path.join(pdfs, '<citekey>.pdf'))
```

Update the `file` field in the just-written entry to `{~/org/pdfs/<citekey>.pdf}`.

---

## Step 5: Annote

Read the abstract (from the entry, or the PDF's opening pages when the work has none).
Write a 2-3 sentence `annote`: plain-language summary written for your own future
reference. The `annote` is the only place for a summary; the site shows it as
"Brief", separate from the verbatim "Abstract".

A good annote answers: **why would I cite this?** It names the specific finding,
claim, or contribution - not just the topic. Prefer concrete over vague:

- Bad: "Discusses the role of automation in decision-making."
- Good: "Shows that automation bias persists even in expert users and cannot be
  eliminated by training - only by redesigning the automation itself."

If the paper is a foundational reference (Simon 1956, Bainbridge 1983), note what
it established and why it remains cited. If it's empirical, name the key result.
If it's a framework or model, say what it lets you do.

Append `annote = {<text>}` to the entry before validating.

---

## Step 6: Audit the abstract

Skip this step if the entry has no `abstract`. Otherwise check it against the
work's own text:

```bash
python3 ~/.agents/skills/citation-add/audit_abstracts.py fetch ~/org/bib/references.bib --keys <citekey>
python3 ~/.agents/skills/citation-add/audit_abstracts.py score ~/org/bib/references.bib --keys <citekey>
```

Act on the result line for the citekey:

- `VERBATIM` - done.
- `REPLACE` - the abstract is not the work's text or is incomplete, and the
  line names a verified source. Write that source's text, then compare it with
  the page: text taken from a PDF (`-> pdf_cut`) can still carry text-layer
  damage such as a mis-scanned quote, so apply the checks under **Abstract**:

  ```bash
  python3 ~/.agents/skills/citation-add/audit_abstracts.py apply ~/org/bib/references.bib --keys <citekey>
  ```

- `DELETE` - the script found no abstract for the work, and the bib text is not
  in the PDF. Check the first page; if the work really has none, run `apply` as
  above to remove the field.
- `REVIEW` - the script cannot settle it: a scanned or column-scrambled PDF, an
  abstract with no heading after it, or sources that disagree. Run
  `audit_abstracts.py show ~/org/bib/references.bib <citekey>` to see the bib
  text next to every source. Compare with the PDF page itself (if the text
  layer is damaged, render it with `pdftoppm -r 110 -f 1 -l 1 -png <pdf> <dir>/page`
  and read the image). Make the bib text match the page exactly, or remove the
  field, then record the check:

  ```bash
  python3 ~/.agents/skills/citation-add/audit_abstracts.py reviewed ~/org/bib/references.bib <citekey> "<what you checked>"
  ```

The audit is a guard, not an oracle. It reliably catches paraphrases, condensed
copies and missing sentences, and it says REVIEW rather than guessing when its
sources are weak. It sends no personal data: a generic User-Agent and no email.

---

## Step 7: Validate after each entry

Run biber after every single entry is added - catch errors immediately, not in bulk:

```bash
biber --tool --validate-datamodel ~/org/bib/references.bib 2>&1 | grep -E "WARN|ERROR"
rm -f ~/org/bib/references_bibertool.bib ~/org/bib/references.bib.blg
```

Fix all ERRORs before moving to the next PDF.
Fix WARNs if possible; note if not.

Target: 0 errors, 0 warnings.

---

## Step 8: Report

After all PDFs are processed:

- Each added citekey: DOI path vs fallback, CrossRef year correction if any
- Abstract source per entry (CrossRef, arXiv, extractor, page read) or why it was omitted
- Audit result per entry, and what was done for REPLACE, DELETE and REVIEW
- Duplicates found and how resolved
- PDFs skipped (NOTEXT, confirmed duplicates)
- Final biber validation status

---

## Re-auditing the whole bib

To check every existing abstract (for example after editing entries by hand),
run `fetch` and `score` without `--keys`. The first full fetch takes a few
minutes; later runs reuse the cache in `~/.cache/citation-add/abstract-audit/`.
`score` prints every entry that is not VERBATIM and writes `report.tsv` there.

- Settle REVIEW rows as in Step 6 and record each with `reviewed`.
- `apply` without `--keys` writes all REPLACE and DELETE results. It refuses to
  run while REVIEW rows remain unless they are covered by an overrides file.
- When the right text needs a source other than the one the script picked,
  put it in an overrides JSON (format in the script's `source_text` docstring:
  `from` is `crossref`, `openalex`, `arxiv`, `cut:N` or `span` with `start`/`end`
  anchors, plus optional `edits` and `drop_from`). Check each with
  `preview <file> <citekey>`, then run `apply --overrides <file>`.

Validate with biber afterwards (Step 7).
# Bibtex Entry Examples

Real cases from this bib. When generating a new entry, find the matching case
and follow it exactly - field names, field order, formatting conventions.

Abstracts are shown as a placeholder to keep the examples short. In a real
entry the `abstract` is the work's own abstract, copied verbatim and complete
(see "Abstract" in SKILL.md), or the field is omitted. Never shorten, reword or
summarize it: the site shows it under an "Abstract" label as the paper's own
text. Summaries belong in `annote`, which the site shows as "Brief".

---

## 1. arXiv preprint

**Situation**: Paper only on arXiv. Filename like `2602.14270v1.pdf` or
`descriptive-name-2602.14270v1.pdf`. DOI is always `10.48550/arXiv.XXXX.XXXXX`.

**Rule**: Use `@article` (our convention). Set `journal` to
`arXiv preprint arXiv:XXXX.XXXXX`. Always include the arXiv DOI.

```bibtex
@article{batista2026sycophantic-ai,
  author    = {Batista, Rafael M. and Griffiths, Thomas L.},
  title     = {A Rational Analysis of the Effects of Sycophantic AI},
  journal   = {arXiv preprint arXiv:2602.14270},
  year      = {2026},
  keywords  = {sycophancy, llm, belief-updating, confirmation-bias, epistemic-risk},
  abstract  = {<the work's own abstract, copied verbatim and complete>},
  file      = {~/org/pdfs/sycophantic-ai-effects-2602.14270v1.pdf},
  doi       = {10.48550/arXiv.2602.14270}
}
```

---

## 2. Journal article (named publisher)

**Situation**: Paper in a named journal published by Elsevier, Nature, Springer,
ACM, etc. The `publisher` field is invalid on `@article` per biber's datamodel.

**Rule**: Use `@article`. Put the publisher in `note = {Publisher: X}` to
preserve the information without triggering a validation warning.

```bibtex
@article{babaei2025nasa-tlx,
  author    = {Babaei, Ebrahim and Dingler, Tilman and Tag, Benjamin and Velloso, Eduardo},
  title     = {Should we use the NASA-TLX in HCI? A review of theoretical and
               methodological issues around Mental Workload Measurement},
  journal   = {International Journal of Human-Computer Studies},
  volume    = {201},
  pages     = {103515},
  year      = {2025},
  note      = {Publisher: Elsevier},
  keywords  = {mental-workload, nasa-tlx, hci, cognitive-load, measurement},
  abstract  = {<the work's own abstract, copied verbatim and complete>},
  file      = {~/org/pdfs/1-s2.0-S1071581925000722-main.pdf},
  doi       = {10.1016/j.ijhcs.2025.103515}
}
```

Filename heuristics: `1-s2.0-S...` = Elsevier ScienceDirect,
`s41597-...` = Nature/Springer.

---

## 3. IEEE conference paper

**Situation**: Paper in IEEE proceedings (ICSA, ICSE, ICSME, etc.). DOI
typically in the footer of page 1. Filename often descriptive.

**Rule**: Use `@inproceedings`. `publisher = {IEEE}` is valid here. Include
full conference name in `booktitle`.

```bibtex
@inproceedings{ivers2025genai-architecture,
  author    = {Ivers, James and Ozkaya, Ipek},
  title     = {Will Generative AI Fill the Automation Gap in Software Architecting?},
  booktitle = {2025 IEEE 22nd International Conference on Software Architecture
               Companion (ICSA-C)},
  year      = {2025},
  pages     = {41--48},
  publisher = {IEEE},
  keywords  = {software-architecture, genai, llm, automation, architecture-decisions},
  abstract  = {<the work's own abstract, copied verbatim and complete>},
  file      = {~/org/pdfs/Will_Generative_AI_Fill_the_Automation_Gap_in_Software_Architecting.pdf},
  doi       = {10.1109/ICSA-C65153.2025.00014}
}
```

---

## 4. ACM conference paper

**Situation**: Paper in ACM proceedings (FAccT, CHI, PLDI, etc.). Filename may
be an ACM DL identifier like `3531146.3533088.pdf`.

**Rule**: Use `@inproceedings`. `publisher = {ACM}` is valid. Use full
proceedings title in `booktitle`.

```bibtex
@inproceedings{weidinger2022taxonomy,
  author    = {Weidinger, Laura and Uesato, Jonathan and Rauh, Maribeth and
               Griffin, Conor and Huang, Po-Sen and Mellor, John and ...},
  title     = {Taxonomy of Risks posed by Language Models},
  booktitle = {Proceedings of the 2022 ACM Conference on Fairness,
               Accountability, and Transparency},
  year      = {2022},
  publisher = {ACM},
  keywords  = {language-models, responsible-ai, risk-taxonomy, safety, ethics},
  abstract  = {<the work's own abstract, copied verbatim and complete>},
  file      = {~/org/pdfs/3531146.3533088.pdf},
  doi       = {10.1145/3531146.3533088}
}
```

---

## 5. OpenReview / ICLR / NeurIPS workshop paper

**Situation**: Paper at ICLR, NeurIPS, ICML, or similar ML venue that
publishes via OpenReview. No traditional publisher. DOI may be an arXiv DOI
if only an arXiv version is available.

**Rule**: Use `@inproceedings`. Omit `publisher` entirely - these venues have
no publisher in the traditional sense. Use the conference short name (e.g.
`ICLR 2025`) in `booktitle` if the full proceedings title is not on the PDF.

```bibtex
@inproceedings{sharma2024sycophancy,
  author    = {Sharma, Mrinank and Tong, Meg and Korbak, Tomasz and ...},
  title     = {Towards Understanding Sycophancy in Language Models},
  booktitle = {ICLR 2024},
  year      = {2024},
  keywords  = {sycophancy, llm, rlhf, human-feedback, alignment},
  abstract  = {<the work's own abstract, copied verbatim and complete>},
  file      = {~/org/pdfs/2310.13548v4.pdf},
  doi       = {10.48550/arXiv.2310.13548}
}
```

---

## 6. SSRN preprint

**Situation**: Paper on SSRN. Filename like `ssrn-XXXXXXX.pdf`. Version
string like `v20260111` may appear on the title page. No DOI on first page.

**Rule**: Use `@misc`. No `publisher`. Omit `doi` if not found on any page
read - do not guess it.

```bibtex
@misc{shaw2026cognitive-surrender,
  author    = {Shaw, Steven D. and Nave, Gideon},
  title     = {Thinking---Fast, Slow, and Artificial: How AI is Reshaping
               Human Reasoning and the Rise of Cognitive Surrender},
  year      = {2026},
  keywords  = {cognitive-surrender, dual-process, human-ai-interaction,
               reasoning, tri-system-theory},
  abstract  = {<the work's own abstract, copied verbatim and complete>},
  file      = {~/org/pdfs/ssrn-6097646.pdf}
}
```

---

## 7. Never invent a DOI

**Situation**: DOI field not visible on page 1 or last page.

**Rule**: Omit the `doi` field entirely. A missing DOI is harmless. A
hallucinated DOI (e.g. `10.1109/XXX.0000.0000000`) silently breaks citation
linking and is worse than no DOI.

Wrong:
```bibtex
  doi = {10.1109/XXX.0000.0000000}   % placeholder - DO NOT DO THIS
  doi = {10.1145/unknown}             % guessed - DO NOT DO THIS
```

Correct: omit the field if unsure.

---

## 8. Book (commercial, no abstract)

**Situation**: A commercially published book (Addison-Wesley, O'Reilly, MIT Press, etc.).
No abstract field - books don't have abstracts. ISBN goes in `note`.

**Rule**: Use `@book`. Include `publisher`, `year`, `keywords`, `note` (for ISBN), `file`.
Omit `abstract` entirely. Omit `doi` unless a DOI is printed in the book itself.

```bibtex
@book{humble2010continuous-delivery,
  author    = {Humble, Jez and Farley, David},
  title     = {Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation},
  publisher = {Addison-Wesley},
  year      = {2010},
  keywords  = {continuous-delivery, continuous-integration, deployment, build-automation, devops},
  note      = {ISBN: 978-0-321-60191-9},
  file      = {~/org/pdfs/Continuous Delivery_ Reliable Software Releases through -- Humble, Jez & Farley, David.pdf}
}
```

---

## 9. Magazine article (no abstract)

**Situation**: Article in a popular-science or trade magazine (Scientific American,
IEEE Spectrum, Communications of the ACM magazine section, etc.). No formal abstract
printed - the article just begins. No DOI visible on pages read.

**Rule**: Use `@article`. Set `journal` to the magazine name as printed. Include
`volume`, `number`, `pages`, `year`. Omit `abstract` entirely - do not summarize
or paraphrase. Omit `doi` if not found on pages read.

```bibtex
@article{schwartz2004tyranny-choice,
  author    = {Schwartz, Barry},
  title     = {The Tyranny of Choice},
  journal   = {Scientific American},
  volume    = {92},
  number    = {4},
  pages     = {70--75},
  year      = {2004},
  keywords  = {choice, decision-making, cognitive-load, paradox-of-choice, well-being},
  file      = {~/org/pdfs/schwartz2004tyranny-of-choice.pdf}
}
```

---

## 10. Non-journal in `journal` field

**Situation**: The `journal` field contains something that is clearly not a
journal - a company name, a personal website, an institution name, etc. (e.g.
`journal = {Lingenic LLC}`).

**Rule**: Reconsider the `@type`. This is likely `@techreport` (if from a
named institution) or `@misc`. Move the institution to the `institution` field
for `@techreport`, or to `note` for `@misc`.

```bibtex
@misc{slavenskoj2026knuth-pseudocode,
  author      = {Slavenskoj, Danslav},
  title       = {Towards a Generalization of Knuth's Pseudocode Architecture:
                 From Algorithms to Knowledge},
  year        = {2026},
  note        = {Lingenic LLC},
  keywords    = {pseudocode, knowledge-representation, formal-systems, notation, llm},
  abstract    = {<the work's own abstract, copied verbatim and complete>},
  file        = {~/org/pdfs/knuth-pseudocode-generalization.pdf}
}
```
# Abstract Extraction Patterns

This file documents which patterns work for which PDF types, and records failures.
When `extract_abstract.py` fails on a new PDF layout, diagnose the failure,
add a new pattern to `extract_abstract.py`, and append a record here.

---

## How to self-heal a FAIL

1. Run `pdftotext -f 1 -l 2 <pdf> -` and inspect the raw text around the abstract
2. Identify why no pattern matched:
   - Different delimiter? (e.g. "Abstract." instead of "Abstract—")
   - Abstract starts on page 3?
   - No "Abstract" keyword at all (structured differently)?
3. Add a new pattern to the `patterns` list in `extract_abstract.py`
   - Give it a descriptive name
   - Keep it before the fallback patterns
   - Test it: `python3 extract_abstract.py <pdf>` should print `ABSTRACT:...`
4. Append a record below: pattern name, what it fixes, example PDF

---

## Pattern records

### ieee_abstract_dash
- **Fixes**: IEEE/ACM two-column papers with "Abstract—" delimiter
- **Terminator**: Index Terms, Keywords, or "I. INTRODUCTION"
- **Added**: 2026-04-09
- **Tested on**: Detecting_Encryption_Vulnerabilities_..., Exploratory_Study_Architectural_Smell_..., Graph_Convolutional_Networks_...

### heading_abstract
- **Fixes**: Papers where "Abstract" is a standalone heading followed by body text
- **Terminator**: Keywords, numbered section heading
- **Added**: 2026-04-09

### caps_abstract
- **Fixes**: Papers using all-caps "ABSTRACT" as heading
- **Terminator**: KEYWORDS, INTRODUCTION
- **Added**: 2026-04-09

### abstract_colon
- **Fixes**: Papers using "Abstract:" with colon
- **Terminator**: Keywords, numbered section
- **Added**: 2026-04-09

### arxiv_blank_lines
- **Fixes**: arXiv preprints where abstract is a blank-line-delimited block
- **Terminator**: double newline after abstract body
- **Added**: 2026-04-09

---

## Known NOABSTRACT types

- **Books**: 2+ signals from: "all rights reserved", "ISBN:", "Library of Congress",
  "Printed in", publisher name, "Draft copy", "Copyright YYYY by"
- **Magazine articles**: Scientific American, IEEE Spectrum, ACM Queue, CACM magazine
  section - no "Abstract" keyword present

---

### iclr_a_bstract
- **Fixes**: ICLR/NeurIPS papers where pdftotext column-split renders "ABSTRACT" as "A BSTRACT"
- **Terminator**: `I NTRODUCTION` (same split artifact) or double newline
- **Added**: 2026-04-09
- **Tested on**: llm-internal-instruction-following-2410.14516v5.pdf, 2310.13548v4.pdf, ssrn-6372119.pdf

### mdpi_abstract_after_info
- **Fixes**: MDPI papers where "Abstract" heading is followed by article-info sidebar (Academic Editors, dates) before the actual abstract text
- **Terminator**: double newline or `Keywords:`
- **Added**: 2026-04-09
- **Tested on**: architecture-05-00094.pdf (yiannoudes2025genai)

### abstract_dot
- **Fixes**: ECSA/Springer LNCS papers using "Abstract." (period) delimiter
- **Terminator**: `Keywords:` or double newline
- **Added**: 2026-04-09
- **Tested on**: adr-in-practice-2024_ECSA.pdf (ahmeti2024adr)

### abstract_space_dash
- **Fixes**: Conference proceedings using "Abstract - " (space-dash) format
- **Terminator**: `Keywords - ` or double newline
- **Added**: 2026-04-09
- **Tested on**: AECTIC-118.pdf (konda2025agentic)

### caps_abstract_inline
- **Fixes**: Papers where "ABSTRACT" appears on same line as abstract text (ACM/IEEE single-column)
- **Terminator**: `CCS Concepts:`, `Keywords:`, or double newline
- **Added**: 2026-04-09
- **Tested on**: Explainability_in_Software_Architectural_Decisions_The_ADR-E_Framework_and_Empirical_Evaluation.pdf

### elsevier_spaced_abstract
- **Fixes**: Elsevier sidebar format rendering as "A B S T R A C T\n\nKeywords:\n<kw>\n...\n\n<abstract>"
- **Terminator**: double newline or end of string
- **Added**: 2026-04-09
- **Tested on**: 1-s2.0-S0090261625000749-main.pdf (shinkle2026scenario-analysis-ai)

### acm_no_heading_ccs
- **Fixes**: ACM preprints with no explicit Abstract heading; abstract precedes CCS Concepts block
- **Terminator**: `CCS Concepts:` or `Additional Key Words`
- **Added**: 2026-04-09
- **Note**: Requires copyright boilerplate (`permission`) before the abstract to anchor the pattern

### Wide page range fallback (pages 1-15)
- **Fixes**: SEI technical reports (ATAM series) and PhD theses where abstract appears on pages 9-11
- **Added**: 2026-04-09
- **Tested on**: 1998_005_001_16646.pdf (kazman1998architecture), 2000_005_001_13706.pdf, LakshithadeSilvaPhDThesis.pdf

### finditer instead of search
- **Fixes**: Documents where first Abstract match is a TOC entry or graphical abstract caption
- **How**: Try all matches for each pattern, return first with length >= 150 chars
- **Added**: 2026-04-09
- **Tested on**: 2000_005_001_13706.pdf (TOC Abstract at pos 2474 → real abstract at pos 5339)

---

## Known NOABSTRACT types (updated)

- **Books**: 2+ signals from: "all rights reserved", "ISBN:", "Library of Congress", "Printed in", publisher name, "Draft copy", "Copyright YYYY by", "Table of Contents"
- **Magazine articles**: Scientific American, IEEE Spectrum, ACM Queue, CACM magazine section
- **Note**: Book/magazine detection now runs AFTER pattern matching, not before. This prevents false positives when publisher names appear in article metadata (e.g., Emerald articles mentioning Springer in related-articles sections).

### elsevier_spaced_abstract_kw_before
- **Fixes**: Elsevier papers where Keywords appear BEFORE "a b s t r a c t" in the sidebar (vs. after in the original elsevier_spaced_abstract pattern)
- **Terminator**: double newline or `1. Introduction`
- **Added**: 2026-04-09
- **Tested on**: 1-s2.0-S0164121222002369-main.pdf (razavian2022odaks)

### acm_ccs_inline
- **Fixes**: ACM/arXiv preprints where abstract is inline before "CCS Concepts:" with no permission block anchor
- **Note**: May capture author affiliation lines before abstract; abstract paragraph is normalized in
- **Added**: 2026-04-09
- **Tested on**: llm-gen-design-rationale-2504.20781v3.pdf (zhou2025llm-design-rationale), 2404.04834v4.pdf (he2025llm-mas-swe)

### nature_background_summary
- **Fixes**: Nature Scientific Data / similar journals where abstract is inline text before "Background & Summary" with no Abstract heading
- **Note**: May capture sidebar/metadata text before the abstract paragraph
- **Added**: 2026-04-09
- **Tested on**: s41597-023-02202-2.pdf (rosenman2023race-ethnicity-names)

---

## Failure log

2026-04-09 | Graph_Convolutional_Networks_Mapping_Source_Code_Architectural_Modules.pdf | ieee_abstract_dash terminator regex `I\.\s+INTRODUCTION` didn't match pdftotext output `I. I NTRODUCTION` (column-layout word split) | Fixed: changed to `I\.\s+I.?NTRODUCTION` to tolerate the split

2026-04-09 | ai-math-research-assistant-2602.22842v1.pdf | ieee_abstract_dash didn't terminate - no Index Terms/Keywords/Introduction heading; abstract ends with blank line before body text | Fixed: added `\n\n` as additional terminator to ieee_abstract_dash

2026-04-09 | adr-drafting-llm-2504.08207v1.pdf | heading_abstract matched graphical abstract caption (50 chars exactly) instead of real abstract | Fixed: increased minimum sanity-check length from 50 to 150 chars

2026-04-09 | grey-systems-Abriefintroduction.pdf | book detection triggered (Emerald article mentions "Springer" in related-articles section) before pattern matching | Fixed: moved book/magazine detection after pattern matching

2026-04-09 | architecture-05-00094.pdf (MDPI) | heading_abstract captured article-info sidebar (Academic Editors, dates) instead of abstract text | Fixed: added mdpi_abstract_after_info pattern before heading_abstract

2026-04-09 | 2001_004_001_13808.pdf | Font-encoded TOC - heading "Abstract" renders as garbled `$EVWUDFW\x03` in pdftotext | Status: genuine FAIL - requires visual read fallback

2026-04-09 | parnas1994software-aging.pdf | pdftotext returns empty text (encrypted/image-only PDF) | Status: genuine FAIL - requires visual read fallback

2026-04-10 | 2212.13179v5.pdf | Springer two-column preprint where author affiliations are interleaved into the abstract block by pdftotext linearization; heading_abstract(wide) matched a TOC "Study title, abstract, keywords" entry instead | Status: genuine FAIL (layout artifact) - abstract extracted manually by filtering affiliation lines from raw text

### abstract_titlecase_inline
- **Fixes**: Springer/arXiv preprints where "Abstract" is title-case and immediately followed by abstract text on the same line (no newline between heading and body)
- **Terminator**: `\nKeywords`, double newline, `\n\d+ Introduction`, or author email block (`\nE-mail:`, `\n\S+ · \S+`)
- **Added**: 2026-04-10
- **Tested on**: 2212.13179v5.pdf (musengamana2024mining) - note: still fails due to interleaved affiliations; visual fallback used for that PDF

---

## Where the abstract is, by document class

Page ranges `classify_pdf.py --task extract:abstract` returns, and what was
found in them. Carried over from the retired cite-nav skill.

#### conference-paper
- Range 1-2 is sufficient for all IEEE/ACM/ECSA conference papers tested (8 entries, all PAGE:1)
- Abstract always on p.1 regardless of delimiter style (Abstract—, Abstract., Abstract -, ABSTRACT inline)
- Notable: ECSA/Springer LNCS uses "Abstract." (period); IEEE two-column splits "INTRODUCTION" as "I NTRODUCTION"

Entries: ivers2025genai-architecture, weidinger2022taxonomy, reiche2025detecting, pandini2025exploratory,
jabeen2025graph, ahmeti2024adr, konda2025agentic, gacitua2026adr

#### journal-article
- Range 1-3 is sufficient for all Elsevier/MDPI/Nature journal articles tested (5 entries, all PAGE:1)
- Elsevier sidebar format renders keywords before or after "a b s t r a c t" - both handled by patterns
- MDPI inserts article-info sidebar (Academic Editors, dates) between "Abstract" heading and text
- Nature Scientific Data has no "Abstract" heading; uses "Background & Summary" instead

Entries: babaei2025nasa-tlx, razavian2022odaks, shinkle2026scenario-analysis-ai,
yiannoudes2025genai, rosenman2023race-ethnicity-names

#### arxiv-preprint
- Range 1-2 is sufficient for all arXiv papers tested (5 entries, all PAGE:1)
- ICLR/NeurIPS column layout splits "ABSTRACT" as "A BSTRACT" in pdftotext - handled by iclr_a_bstract pattern
- ACM/arXiv preprints: abstract inline before "CCS Concepts:" with no explicit heading

Entries: batista2026sycophantic-ai, sharma2024sycophancy, heo2025llm-instruction-following,
zhou2025llm-design-rationale, he2025llm-mas-swe

#### ssrn-preprint
- Range 1-2 sufficient; abstract on p.1 in all cases tested

Entries: shaw2026cognitive-surrender, kim2026harness-engineering

#### tech-report
- Range 1-15 required; SEI/CMU reports have front matter (cover, ToC, executive summary) before abstract
- Abstract appears on pp.9-11 for ATAM series reports
- First "Abstract" match in pdftotext may be a TOC entry - finditer needed to skip short matches

Entries: kazman1998architecture (PAGE:9), kazman2000atam (PAGE:9)

#### thesis
- Range 1-15 sufficient via toc-first strategy; abstract typically pp.3-5

Entries: desilva2012phd-arch-erosion (PAGE:3)

#### book
- NOTEXT for all books tested (scanned/encrypted); extract:abstract not applicable to books anyway

---

## Known page-range failures

#### Springer two-column preprint with interleaved affiliations
- **Trigger**: `2212.13179v5.pdf` - author affiliations interleaved into abstract block by pdftotext linearization;
  heading_abstract(wide) matched a TOC "Study title, abstract, keywords" entry instead
- **Status**: genuine layout artifact; abstract_titlecase_inline pattern added but still fails for this PDF;
  visual fallback required
- **Date**: 2026-04-10
# classify_pdf.py: classification log

Read this when `classify_pdf.py` puts a document in the wrong class.
Records edge cases where its deterministic rules were wrong or surprising.
Routine correct classifications are not logged here - only failures, fixes, and boundary cases.

New entries go at the bottom of the relevant doc-class section.

---

## Known failure modes and fixes

### arXiv papers with descriptive filenames misclassified as thesis
- **Trigger**: arXiv paper saved with descriptive filename (e.g. `sycophantic-ai-effects-2602.14270v1.pdf`);
  university dept names in affiliations (e.g. "Department of Psychology") matched `master` substring in thesis regex
- **Fix**: moved arXiv check (`arxiv:` in text + `\d{4}\.\d{4,5}v\d` in filename) before thesis check;
  tightened thesis regex to `\b(phd|ph\.d\.?|doctoral|master\'?s thesis|dissertation)\b`
- **Date**: 2026-04-10

### SSRN preprints classified as unknown
- **Trigger**: `bib-type=misc` not mapped; no strong venue signals in SSRN page-1 text
- **Fix**: added `ssrn-` filename prefix check and `social science research network` text signal
- **Date**: 2026-04-10

### inproceedings early-return masked arXiv classification
- **Trigger**: ICLR papers submitted as `bib-type=inproceedings` returned conference-paper before arXiv check ran
- **Fix**: moved inproceedings early-return to after arXiv check in classify()
- **Date**: 2026-04-10

### ACM papers regressed to book after inproceedings fix
- **Trigger**: removing inproceedings early-return let ACM DL download metadata ("all rights reserved", publisher names)
  trigger book text signal check
- **Fix**: changed book detection to use page count only (>150pp), removed text signals for book class
- **Date**: 2026-04-10

### Nature/MDPI journals misclassified as conference-paper
- **Trigger**: Nature Scientific Data lacks "journal"/"volume"/"issue" on p.1; ~15pp falls into conference fallback
- **Fix**: added `nature\.com|mdpi\.com|tandfonline|sage pub` to journal-article regex
- **Date**: 2026-04-10

---

## Boundary cases by doc-class

### arxiv-preprint
- `batista2026sycophantic-ai`: descriptive filename triggered thesis before fix; arXiv: in text is reliable signal
- `heo2025llm-instruction-following`: bib-type=inproceedings but arxiv: signal takes priority (correct)
- `sharma2024sycophancy`: ICLR via arXiv; classified arxiv-preprint despite inproceedings bib-type

### ssrn-preprint
- `shaw2026cognitive-surrender`: ssrn-XXXXXXX.pdf filename is reliable; was unknown before fix
- `kim2026harness-engineering`: same pattern confirmed

### journal-article
- `gacitua2026adr`: bib-type=article but no strong journal signals on p.1; falls through to conference-paper
  (acceptable - range is still correct for abstract extraction)
- `rosenman2023race-ethnicity-names`: Nature Scientific Data; fixed by nature\.com signal
#!/usr/bin/env python3
"""
Fetch structured metadata from CrossRef API by DOI.

Usage:
    python3 fetch_doi.py <doi>

Output (stdout, one per line):
    FIELD:key=value   - metadata fields from CrossRef
    NOTFOUND:<doi>    - DOI not in CrossRef
    ERROR:<reason>    - network or parse error

Fields returned (when available):
    type        - CrossRef work type (journal-article, proceedings-article, etc.)
    title
    author      - "Lastname, Firstname and Lastname, Firstname" format
    year
    journal     - container title (for articles)
    booktitle   - container title (for proceedings)
    volume
    number
    pages
    publisher
    doi
    abstract    - present for many publishers; JATS markup converted to plain text

Inference reads these fields and formats the final BibTeX entry
following examples.md conventions. fetch_doi.py only fetches - it does
not format.

Exit codes:
    0 - success (fields printed)
    1 - not found or error
"""

import sys
import html
import json
import re
import urllib.request
import urllib.error


def fetch_crossref(doi):
    url = f"https://api.crossref.org/works/{urllib.request.quote(doi, safe='/')}"
    req = urllib.request.Request(
        url,
        headers={"User-Agent": "citation-add/1.0"}
    )
    try:
        with urllib.request.urlopen(req, timeout=10) as resp:
            return json.loads(resp.read())
    except urllib.error.HTTPError as e:
        if e.code == 404:
            return None
        raise RuntimeError(f"HTTP {e.code}")
    except Exception as e:
        raise RuntimeError(str(e))


def strip_html(s):
    return re.sub(r"<[^>]+>", "", s or "").strip()


def jats_abstract(s):
    """CrossRef abstracts are JATS XML. Drop a leading "Abstract" heading, keep
    structured-abstract section titles as "Title:", keep paragraph breaks as
    spaces, and decode entities, so the text matches the printed abstract."""
    if not s:
        return ""
    s = re.sub(r"<jats:title>\s*(abstract|summary)\s*</jats:title>", " ", s, flags=re.I)
    s = re.sub(r"<jats:title>\s*([^<]*?)\s*:?\s*</jats:title>", r" \1: ", s)
    s = re.sub(r"</?jats:(p|sec)[^>]*>", " ", s)
    s = html.unescape(strip_html(s))
    s = re.sub(r"\s+", " ", s).strip()
    return re.sub(r"^(abstract|summary)\s*[:.\u2014-]?\s+", "", s, flags=re.I)


def parse_fields(data, doi):
    w = data.get("message", {})
    fields = {}

    fields["type"] = w.get("type", "")

    titles = w.get("title", [])
    if titles:
        fields["title"] = strip_html(titles[0])

    authors = w.get("author", [])
    if authors:
        parts = []
        for a in authors:
            if a.get("family"):
                given = a.get("given", "")
                parts.append(f"{a['family']}, {given}" if given else a["family"])
        if parts:
            fields["author"] = " and ".join(parts)

    for date_field in ("published-print", "published-online", "published", "issued"):
        dp = w.get(date_field, {}).get("date-parts", [[]])
        if dp and dp[0]:
            fields["year"] = str(dp[0][0])
            break

    containers = w.get("container-title", [])
    if containers:
        cr_type = fields.get("type", "")
        if cr_type == "proceedings-article":
            fields["booktitle"] = containers[0]
        else:
            fields["journal"] = containers[0]

    for f in ("volume", "issue", "page", "publisher"):
        v = w.get(f, "")
        if v:
            out_key = "number" if f == "issue" else ("pages" if f == "page" else f)
            fields[out_key] = str(v)

    fields["doi"] = doi

    abstract = jats_abstract(w.get("abstract", ""))
    if abstract:
        fields["abstract"] = abstract

    return fields


def main():
    if len(sys.argv) < 2:
        print("Usage: fetch_doi.py <doi>", file=sys.stderr)
        sys.exit(1)

    doi = sys.argv[1].strip()
    # Normalize - strip URL prefix if present
    doi = re.sub(r"^https?://doi\.org/", "", doi)
    doi = re.sub(r"^doi:\s*", "", doi, flags=re.IGNORECASE)

    try:
        data = fetch_crossref(doi)
    except RuntimeError as e:
        print(f"ERROR:{e}")
        sys.exit(1)

    if data is None:
        print(f"NOTFOUND:{doi}")
        sys.exit(1)

    fields = parse_fields(data, doi)
    for k, v in fields.items():
        # Escape newlines in multi-line values (abstract)
        v_clean = v.replace("\n", " ").replace("\r", "")
        print(f"FIELD:{k}={v_clean}")

    # Warn if CrossRef returned too few fields to build a complete entry
    # (excludes 'type' and 'doi' which are always present)
    substantive = {k for k in fields if k not in ('type', 'doi')}
    if len(substantive) < 4:
        missing = {'title', 'author', 'year', 'journal', 'booktitle'} - substantive
        print(f"SPARSE:CrossRef returned only {len(substantive)} substantive fields. "
              f"Missing: {', '.join(sorted(missing))}. Read PDF pages for fallback.")


if __name__ == "__main__":
    main()
#!/usr/bin/env python3
"""
Deterministically extract abstracts from PDFs using pdftotext + heuristics.

Usage:
    python3 extract_abstract.py <pdf_path>

Output (to stdout):
    ABSTRACT:<text>        - abstract found verbatim
    NOABSTRACT:<reason>    - no abstract expected (book, magazine, etc.)
    FAIL:<reason>          - extraction failed, fall back to inference

Exit codes:
    0 - ABSTRACT found
    1 - NOABSTRACT (not an error, just not applicable)
    2 - FAIL (extraction failed, fall back to inference)

Patterns file:
    patterns.md - documents what patterns work and what failed, for self-healing.
    When a new pattern is added to this script, append a note to patterns.md.
"""

import sys
import os
import re
import subprocess

PDF_PATH = os.path.expanduser(sys.argv[1]) if len(sys.argv) > 1 else None

# --- Helpers ---

def extract_text(pdf_path, pages="1-2"):
    """Run pdftotext on given page range, return text or None."""
    try:
        result = subprocess.run(
            ["pdftotext", "-f", pages.split("-")[0], "-l", pages.split("-")[-1], pdf_path, "-"],
            capture_output=True, text=True, timeout=30
        )
        return result.stdout if result.returncode == 0 else None
    except Exception:
        return None


def normalize(text):
    """Collapse whitespace."""
    return re.sub(r'\s+', ' ', text).strip()


# --- Pattern library ---
# Each pattern is tried in order. First match wins.
# Returns (abstract_text, pattern_name) or None.

def try_patterns(text):
    patterns = [
        # Pattern 1: IEEE/ACM two-column format
        # "Abstract—" at start; terminates at Index Terms, Keywords, Introduction heading,
        # or a blank line (two-column layout where body text follows after blank line).
        # Note: pdftotext sometimes splits "INTRODUCTION" as "I NTRODUCTION" due to column layout.
        ("ieee_abstract_dash",
         r'Abstract[—\-–]\s*(.+?)(?=\n\n|\n\s*(?:Index Terms|Keywords|I\.\s+I.?NTRODUCTION|\d+\.\s+INTRODUCTION))',
         re.DOTALL | re.IGNORECASE),

        # Pattern 2: ICLR/NeurIPS spaced-cap heading "A BSTRACT"
        # pdftotext splits the heading due to column layout: "A BSTRACT\n<text>"
        # Terminates at "I NTRODUCTION" (same split artifact) or blank line.
        ("iclr_a_bstract",
         r'A BSTRACT\s*\n(.+?)(?=\n\n|\n\s*I\s+NTRODUCTION)',
         re.DOTALL),

        # Pattern 2b: MDPI style - "Abstract" heading followed by article-info sidebar,
        # then blank line, then actual abstract text.
        # e.g. "Abstract\n\nAcademic Editors: ...\nReceived: ...\n\n<actual abstract>"
        ("mdpi_abstract_after_info",
         r'Abstract\s*\n\n(?:Academic Editors?:.*?\n|Received:.*?\n|Revised:.*?\n|Accepted:.*?\n)+\n(.+?)(?=\n\n|\nKeywords?:)',
         re.DOTALL | re.IGNORECASE),

        # Pattern 3: "Abstract" as heading, body follows on next line(s)
        # Terminates at Keywords, Index Terms, numbered section, or end of text block.
        # Skip if immediately followed by article metadata (handled by mdpi_abstract_after_info).
        ("heading_abstract",
         r'(?<!\w)Abstract\s*\n\s*(.+?)(?=\n\s*(?:Keywords|Index Terms|1\s+Introduction|\d\.\s+[A-Z])|\n\n)',
         re.DOTALL | re.IGNORECASE),

        # Pattern 4: "ABSTRACT" all caps inline - text follows on same line
        # e.g. "ABSTRACT As software systems..." (ACM/IEEE single-column)
        # Terminates at CCS Concepts, Keywords, or double newline.
        ("caps_abstract_inline",
         r'\bABSTRACT\s+(.+?)(?=\n\s*(?:CCS Concepts|Keywords|Additional Key Words|\d+\s+INTRODUCTION)|\n\n)',
         re.DOTALL),

        # Pattern 4b: "Abstract" title-case inline - text follows on same line
        # e.g. "Abstract Mining Software Repositories..." (Springer/arXiv preprints)
        # Terminates at Keywords, double newline, numbered section, or author email block.
        ("abstract_titlecase_inline",
         r'\bAbstract\s+(.+?)(?=\nKeywords|\n\n|\n\d+\s+Introduction|\n\S+\s+·\s+\S+|\nE-mail:)',
         re.DOTALL),

        # Pattern 5: "ABSTRACT" all caps as heading, text on next line
        ("caps_abstract",
         r'\bABSTRACT\b\s*\n\s*(.+?)(?=\n\s*(?:KEYWORDS|KEY WORDS|INTRODUCTION|Keywords|1\s+\w)|\n\n)',
         re.DOTALL | re.IGNORECASE),

        # Pattern 6: "Abstract:" with colon
        ("abstract_colon",
         r'\bAbstract:\s*(.+?)(?=\n\s*(?:Keywords|Index Terms|1\.\s|\d\.\s+\w)|\n\n)',
         re.DOTALL | re.IGNORECASE),

        # Pattern 7: "Abstract." with period (ECSA/Springer LNCS format)
        # e.g. "Abstract. To establish good architectural practice..."
        # Terminates at Keywords: or next blank line after section ends.
        ("abstract_dot",
         r'\bAbstract\.\s+(.+?)(?=\n\s*(?:Keywords|Index Terms|1\s+Introduction)|\n\n)',
         re.DOTALL | re.IGNORECASE),

        # Pattern 8: "Abstract - " with space-dash (conference proceedings variant)
        # e.g. "Abstract - The maturation of agentic AI..."
        # Terminates at "Keywords - " or double newline.
        ("abstract_space_dash",
         r'\bAbstract\s+-\s+(.+?)(?=\nKeywords\s+-|\n\n)',
         re.DOTALL | re.IGNORECASE),

        # Pattern 9: Elsevier "A R T I C L E I N F O / A B S T R A C T" sidebar format
        # pdftotext renders as "A B S T R A C T\n\nKeywords:\n<kw>\n...\n\n<abstract text>"
        # The keywords sidebar appears first, then a blank line, then the abstract.
        ("elsevier_spaced_abstract",
         r'A B S T R A C T\s*\n\n(?:Keywords:[^\n]*\n(?:[^\n]+\n)*)\n(.+?)(?=\n\n|\Z)',
         re.DOTALL),

        # Pattern 9b: Elsevier sidebar where Keywords appear BEFORE "a b s t r a c t"
        # pdftotext renders article-info + keywords block, then "a b s t r a c t\n<text>"
        # e.g. "Keywords:\n...\n\na b s t r a c t\n<abstract text>\n\n1. Introduction"
        ("elsevier_spaced_abstract_kw_before",
         r'a b s t r a c t\s*\n(.+?)(?=\n\n|\n\s*\d+\.\s+Introduction|\Z)',
         re.DOTALL | re.IGNORECASE),

        # Pattern 10: ACM preprints with no explicit Abstract heading
        # Abstract text follows author affiliations and copyright boilerplate,
        # terminated by "CCS Concepts:" or "Additional Key Words".
        ("acm_no_heading_ccs",
         r'(?:permission[^\n]+\n.{0,500}?)\n\n(.+?)(?=\nCCS Concepts|\nAdditional Key Words)',
         re.DOTALL | re.IGNORECASE),

        # Pattern 10b: ACM/arXiv papers where abstract is inline before "CCS Concepts:"
        # with no explicit Abstract heading and no permission block anchor.
        # The abstract is the last paragraph-length block before "CCS Concepts:".
        # Strategy: find the last \n\n<text>\nCCS Concepts: where text >= 150 chars and
        # contains no \n\n (single paragraph).
        ("acm_ccs_inline",
         r'\n\n([^\n]{20}[^\n]*(?:\n[^\n]+){2,})\nCCS Concepts:',
         re.DOTALL),

        # Pattern 12: Nature Scientific Data / similar journals
        # Abstract is inline text directly after author line, terminated by "Background & Summary"
        # or next section heading. No "Abstract" keyword present.
        ("nature_background_summary",
         r'\n\n(.+?)\n\nBackground\s*&\s*Summary',
         re.DOTALL),

        # Pattern 11: arXiv style - abstract block bounded by blank lines after "Abstract"
        ("arxiv_blank_lines",
         r'\bAbstract\b\s*\n\n(.+?)\n\n',
         re.DOTALL | re.IGNORECASE),

    ]

    for name, pattern, flags in patterns:
        for m in re.finditer(pattern, text, flags):
            candidate = normalize(m.group(1))
            # Sanity checks: must be at least 150 chars, not more than 6000
            # (150 excludes titles and graphical abstract captions)
            if 150 <= len(candidate) <= 6000:
                return candidate, name

    return None


# --- Book/non-abstract detection ---

BOOK_SIGNALS = [
    r'all rights reserved',
    r'isbn[:\s]',
    r'library of congress',
    r'printed in',
    r'addison.?wesley',
    r"o'reilly",
    r'mit press',
    r'springer',
    r'draft copy',
    r'copyright.*\d{4}.*by',
    r'table of contents',
]

MAG_SIGNALS = [
    r'scientific american',
    r'ieee spectrum',
    r'communications of the acm',
    r'acm queue',
]


def is_book(text_lower):
    hits = sum(1 for sig in BOOK_SIGNALS if re.search(sig, text_lower))
    return hits >= 2


def is_magazine(text_lower):
    hits = sum(1 for sig in MAG_SIGNALS if re.search(sig, text_lower))
    return hits >= 1 and not re.search(r'\babstract\b', text_lower)


# --- Entry point ---

def main():
    if not PDF_PATH:
        print("FAIL:no pdf path given")
        sys.exit(2)

    if not os.path.exists(PDF_PATH):
        print(f"FAIL:file not found: {PDF_PATH}")
        sys.exit(2)

    # First pass: pages 1-2
    text = extract_text(PDF_PATH, "1-2")
    if not text or len(text.strip()) < 20:
        print("FAIL:pdftotext returned empty text")
        sys.exit(2)

    # Try patterns before book/magazine detection - patterns take priority
    result = try_patterns(text)
    if result:
        abstract, pattern_name = result
        print(f"ABSTRACT:{abstract}")
        print(f"PATTERN:{pattern_name}", file=sys.stderr)
        sys.exit(0)

    # Second pass: pages 1-15 (catches tech reports, PhD theses with late abstract pages)
    text_wide = extract_text(PDF_PATH, "1-15")
    if text_wide and len(text_wide.strip()) > len(text.strip()):
        result = try_patterns(text_wide)
        if result:
            abstract, pattern_name = result
            print(f"ABSTRACT:{abstract}")
            print(f"PATTERN:{pattern_name}(wide)", file=sys.stderr)
            sys.exit(0)

    # No patterns matched - check if this is a book or magazine (no abstract expected)
    text_lower = text.lower()
    if is_book(text_lower):
        print("NOABSTRACT:appears to be a book (no abstract expected)")
        sys.exit(1)

    if is_magazine(text_lower):
        print("NOABSTRACT:appears to be a magazine article (no formal abstract)")
        sys.exit(1)

    print("FAIL:abstract section not found by any pattern")
    sys.exit(2)


if __name__ == "__main__":
    main()
#!/usr/bin/env python3
"""
Classify a PDF for citation-add: document class, DOI, pdfinfo metadata, and
which pages to read for the bib fields or the abstract.

Copied from the retired cite-nav skill; record classification mistakes and
fixes in classify.md next to this script.

Usage:
    python3 classify_pdf.py --pdf <path> --task <task> [--bib-type <type>] [--hint <hint>]

Tasks:
    classify          - classify document, return CLASS only
    extract:abstract  - return range/strategy for abstract extraction
    extract:bib       - return range/strategy and field list for bib entry extraction

Output (stdout):
    CLASS:<name>           - document class inferred
    RANGE:<start>-<end>    - page range to read (extract tasks only)
    STRATEGY:<name>        - strategy used (extract tasks only)
    FIELDS:<f1,f2,...>     - bib fields to read off the pages (extract:bib only);
                             never abstract, which citation-add copies verbatim
    NOTEXT:<reason>        - pdftotext returned empty (scanned/encrypted)

Exit codes:
    0 - success
    2 - NOTEXT or classification failed
"""

import sys
import os
import re
import subprocess
import argparse

def pdfinfo(pdf_path):
    """Return dict of pdfinfo fields."""
    try:
        result = subprocess.run(['pdfinfo', pdf_path], capture_output=True, text=True, timeout=10)
        info = {}
        for line in result.stdout.splitlines():
            if ':' in line:
                k, _, v = line.partition(':')
                info[k.strip()] = v.strip()
        return info
    except Exception:
        return {}

def page_count(pdf_path):
    info = pdfinfo(pdf_path)
    try:
        return int(info.get('Pages', 0))
    except ValueError:
        return 0

def meta_signals(info):
    """
    Extract classification signals and bib fields from pdfinfo metadata.
    Returns (signals_dict, fields_dict).
      signals_dict: keys that classify() can use (arxiv, thesis, etc.)
      fields_dict:  bib fields found (title, author, keywords)
    """
    title = info.get('Title', '').strip()
    author = info.get('Author', '').strip()
    subject = info.get('Subject', '').strip()
    keywords = info.get('Keywords', '').strip()
    creator = info.get('Creator', '').strip()

    title_lower = title.lower()
    creator_lower = creator.lower()

    signals = {}
    if 'arxiv genpdf' in creator_lower or 'arxiv' in creator_lower:
        signals['arxiv'] = True
    if re.search(r'\b(phd|ph\.d|doctoral|master.s thesis|dissertation)\b', title_lower):
        signals['thesis'] = True
    if subject:
        signals['subject'] = subject.lower()

    fields = {}
    if title:
        fields['title'] = title
    if author:
        fields['author'] = author
    if keywords:
        fields['keywords'] = keywords

    return signals, fields

def extract_doi(pdf_path, info=None, text=None):
    """
    Extract DOI from pdfinfo metadata or p.1 text.
    Only matches DOIs that appear near a doi: anchor to avoid picking up
    reference list DOIs.
    Returns DOI string or None.
    """
    bare_doi = re.compile(r'\b10\.\d{4,9}/[^\s,;>"\])\}]+')
    # Anchored: doi: or doi.org/ prefix - these are the paper's own DOI
    anchored_doi = re.compile(
        r'(?:doi\.org/|doi:\s*|DOI:\s*)(10\.\d{4,9}/[^\s,;>"\])\}]+)',
        re.IGNORECASE
    )

    # Check pdfinfo fields first (always reliable)
    if info:
        for field in ('doi', 'DOI', 'Subject', 'Keywords'):
            val = info.get(field, '')
            m = bare_doi.search(val)
            if m:
                return m.group(0).rstrip('.')

    # Scan p.1 text - anchored only to avoid reference DOIs
    if text is None:
        text = extract_text(pdf_path, "1-1") or ""
    m = anchored_doi.search(text)
    if m:
        doi = m.group(1).rstrip('.')
        # Filter placeholder DOIs (e.g. ACM template 10.1145/nnnnnnn)
        if not re.search(r'[nN]{3,}', doi):
            return doi

    return None


def extract_text(pdf_path, pages="1-2"):
    try:
        start, end = pages.split('-')
        result = subprocess.run(
            ['pdftotext', '-f', start, '-l', end, pdf_path, '-'],
            capture_output=True, text=True, timeout=30
        )
        return result.stdout if result.returncode == 0 else None
    except Exception:
        return None

def classify(pdf_path, bib_type=None, hint=None, info=None):
    """
    Classify document into one of:
      journal-article, conference-paper, arxiv-preprint,
      book, thesis, tech-report, unknown

    Priority: bib-type arg > pdfinfo metadata signals > filename > p.1 text > page count
    """
    # Bib type mapping - resolve unambiguous types immediately
    if bib_type:
        t = bib_type.lower()
        if t == 'book':
            return 'book'
        if t in ('mastersthesis', 'phdthesis'):
            return 'thesis'
        if t == 'techreport':
            return 'tech-report'
        # inproceedings, article, misc: refine below (may be arxiv-preprint)

    if info is None:
        info = pdfinfo(pdf_path)
    meta, _ = meta_signals(info)

    # pdfinfo metadata signals - cheap, before reading any page text
    if meta.get('arxiv'):
        return 'arxiv-preprint'
    if meta.get('thesis'):
        return 'thesis'

    pages = page_count(pdf_path) if not info else int(info.get('Pages', 0) or 0)
    text = extract_text(pdf_path, "1-1") or ""
    text_lower = text.lower()
    filename = os.path.basename(pdf_path).lower()

    # arXiv signals - check early, before all other heuristics.
    # arXiv papers mention university affiliations which falsely trigger thesis,
    # and "Published as a conference paper at ICLR" which triggers conference.
    if (re.search(r'arxiv:', text_lower)
            or re.search(r'\d{4}\.\d{4,5}v\d', filename)
            or 'arxiv' in (hint or '').lower()):
        return 'arxiv-preprint'

    # SSRN signals
    if (re.search(r'ssrn-\d+', filename)
            or re.search(r'ssrn|social science research network', text_lower)):
        return 'ssrn-preprint'

    # inproceedings early return - safe now that arXiv has been checked
    if bib_type and bib_type.lower() == 'inproceedings':
        return 'conference-paper'

    # Book signals - page count is the primary signal; text signals alone are
    # too noisy (ACM DL downloads embed "all rights reserved" in metadata)
    if pages > 150:
        return 'book'

    # Thesis signals - require strong explicit markers, not just substrings
    # ('master' alone matches 'masterclass', 'masterful', university dept names)
    if re.search(r'\b(phd|ph\.d\.?|doctoral|master\'?s thesis|dissertation)\b', text_lower):
        return 'thesis'

    # Tech report signals
    if re.search(r'technical report|cmu/sei|sei/cmu|nist |rand corporation', text_lower):
        return 'tech-report'

    # Conference vs journal
    if re.search(r'proceedings|conference|workshop|symposium|icse|esec|fse|ecsa|icsa', text_lower):
        return 'conference-paper'

    if re.search(r'journal|volume|issue|elsevier|springer|wiley|ieee trans|nature\.com|mdpi\.com|tandfonline|sage pub', text_lower):
        return 'journal-article'

    # Fallback by page count
    if pages <= 12:
        return 'conference-paper'
    if pages <= 30:
        return 'journal-article'

    return 'unknown'


def strategy_for(doc_class, task, pages):
    """
    Return (strategy_name, page_range_string) for a doc class + task.

    Strategies:
      full-read    - read the whole document
      front-read   - read first N pages (abstract always here)
      toc-first    - read TOC pages, then navigate
      term-narrow  - read second half of document
    """
    # Normalize task aliases
    if task == 'abstract':
        task = 'extract:abstract'

    if task == 'extract:abstract':
        if doc_class in ('conference-paper', 'arxiv-preprint', 'ssrn-preprint'):
            return 'front-read', '1-2'
        if doc_class == 'journal-article':
            return 'front-read', '1-3'
        if doc_class == 'tech-report':
            return 'front-read', '1-15'   # SEI reports abstract on p9-11
        if doc_class in ('book', 'thesis'):
            return 'toc-first', f'1-{min(15, pages)}'
        return 'front-read', '1-3'

    if task == 'extract:bib':
        # Bib extraction needs the same range as abstract plus p.1 for metadata
        # For short docs, full-read is cheap enough
        if pages <= 30:
            return 'full-read', f'1-{pages}'
        if doc_class in ('book', 'thesis'):
            return 'toc-first', f'1-{min(15, pages)}'
        return 'front-read', '1-5'

    return 'full-read', f'1-{min(pages, 20)}'


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--pdf', required=True)
    parser.add_argument('--task', required=True)
    parser.add_argument('--bib-type', default=None)
    parser.add_argument('--hint', default=None)
    args = parser.parse_args()

    pdf_path = os.path.expanduser(args.pdf)
    if not os.path.exists(pdf_path):
        print(f"NOTEXT:file not found: {pdf_path}")
        sys.exit(2)

    # Quick text check
    text = extract_text(pdf_path, "1-1")
    if not text or len(text.strip()) < 20:
        print("NOTEXT:pdftotext returned empty (scanned or encrypted)")
        sys.exit(2)

    info = pdfinfo(pdf_path)
    pages = int(info.get('Pages', 0) or 0) or 1
    doc_class = classify(pdf_path, bib_type=args.bib_type, hint=args.hint, info=info)

    print(f"CLASS:{doc_class}")

    if args.task == 'classify':
        return

    strategy, page_range = strategy_for(doc_class, args.task, pages)
    print(f"RANGE:{page_range}")
    print(f"STRATEGY:{strategy}")

    if args.task == 'extract:bib':
        # Emit metadata fields found directly from pdfinfo
        _, meta_fields = meta_signals(info)
        for k, v in meta_fields.items():
            print(f"META:{k}={v}")
        # DOI - try pdfinfo then p.1 text
        doi = extract_doi(pdf_path, info=info)
        if doi:
            print(f"DOI:{doi}")
        # Fields still to read off the pages (those not in metadata or doi).
        # The abstract is not one of them: citation-add copies it verbatim from
        # CrossRef, arXiv or the PDF, or omits it.
        needed = [f for f in ('author', 'title', 'year', 'venue')
                  if f not in meta_fields]
        if not doi and doc_class not in ('book', 'thesis'):
            needed.append('doi')
        print(f"FIELDS:{','.join(needed)}")


if __name__ == '__main__':
    main()
#!/usr/bin/env python3
"""
Duplicate detection for citation-add.

Given a new PDF's extracted DOI and/or title, checks references.bib for
existing entries that may be the same work. Outputs structured instructions
for the calling inference agent.

Usage:
    python3 check_duplicate.py --bib ~/org/bib/references.bib
                               [--doi <doi>]
                               [--title <title>]
                               [--author <author>]
                               [--year <year>]
                               [--new-pdf <filename>]

Output (stdout, one per line):
    NO_MATCH
        No existing entry looks like this paper. Proceed to add.

    KEEP_EXISTING: citekey=<key> existing_file=<path> reason=<text>
        Deterministic: delete new PDF, keep existing. Reason explains why.

    KEEP_NEW: citekey=<key> existing_file=<path> reason=<text>
        Deterministic: update bib file field to new PDF, delete old PDF.

    INFER: citekey=<key> existing_file=<path> new_pdf=<filename> hint=<text>
        Read pages 1-2 of both PDFs and reason about whether they are the
        same work. Resolve as KEEP_EXISTING, KEEP_NEW, or ASK based on
        content. If resolved silently, log the decision in the run report.

    ASK: citekey=<key> existing_file=<path> message=<text>
        Genuinely ambiguous. Show message to user and wait for input.

Exit codes:
    0 - output produced (check first token to interpret)
    1 - usage error
"""

import sys
import re
import argparse


ARXIV_RE = re.compile(r'(\d{4}\.\d{4,5})(?:v(\d+))?(?:\.pdf)?$', re.IGNORECASE)


def extract_field(name, block):
    """Extract a bib field value, handling nested braces correctly."""
    m = re.search(r'\b' + name + r'\s*=\s*\{', block, re.IGNORECASE)
    if not m:
        return ''
    start = m.end()
    depth = 1
    i = start
    while i < len(block) and depth > 0:
        if block[i] == '{':
            depth += 1
        elif block[i] == '}':
            depth -= 1
        i += 1
    return block[start:i - 1].strip()


def load_bib(bib_path):
    """Parse bib into list of dicts with keys: citekey, doi, title, author, year, file."""
    entries = []
    try:
        text = open(bib_path, encoding='utf-8').read()
    except FileNotFoundError:
        return entries

    for block in re.split(r'(?=@\w+\s*\{)', text):
        block = block.strip()
        if not block.startswith('@'):
            continue
        m = re.match(r'@\w+\s*\{([^,]+),', block)
        if not m:
            continue
        citekey = m.group(1).strip()

        entries.append({
            'citekey': citekey,
            'doi': extract_field('doi', block).lower().strip(),
            'title': extract_field('title', block).lower().strip(),
            'author': extract_field('author', block),
            'year': extract_field('year', block).strip(),
            'file': extract_field('file', block),
        })
    return entries


def normalize_doi(doi):
    if not doi:
        return ''
    doi = re.sub(r'^https?://doi\.org/', '', doi, flags=re.IGNORECASE)
    doi = re.sub(r'^doi:\s*', '', doi, flags=re.IGNORECASE)
    return doi.lower().strip()


def title_words(title):
    """Return set of significant lowercase words from a title."""
    stop = {'a', 'an', 'the', 'of', 'in', 'on', 'for', 'and', 'or', 'to',
            'with', 'using', 'towards', 'combining', 'via'}
    words = re.findall(r'[a-z0-9]+', (title or '').lower())
    return {w for w in words if w not in stop and len(w) > 2}


def author_surnames(author_str):
    """Return set of lowercase surnames from a bib author string.

    Handles both 'Lastname, Firstname and ...' and 'Firstname Lastname and ...' forms.
    Also handles raw nav.py output like 'John Smith; Jane Doe'.
    """
    if not author_str:
        return set()
    # Split on ' and ' or ';'
    parts = re.split(r'\s+and\s+|;', author_str, flags=re.IGNORECASE)
    surnames = set()
    for part in parts:
        part = part.strip()
        if not part:
            continue
        # 'Lastname, Firstname' form
        if ',' in part:
            surname = part.split(',')[0].strip()
        else:
            # 'Firstname Lastname' - take last token
            tokens = part.split()
            surname = tokens[-1] if tokens else ''
        # Strip LaTeX accents like {\v{c}} -> just take alpha chars
        surname = re.sub(r'\{[^}]*\}', '', surname)
        surname = re.sub(r'[^a-zA-Z]', '', surname).lower()
        if surname:
            surnames.add(surname)
    return surnames


def arxiv_id_and_version(filename):
    """Return (arxiv_id, version_int) or (None, None)."""
    m = ARXIV_RE.search(filename or '')
    if m:
        arxiv_id = m.group(1)
        version = int(m.group(2)) if m.group(2) else 0
        return arxiv_id, version
    return None, None


def signals_match(new_title, new_author, new_year, entry, title_threshold):
    """Check whether title, author, and year all agree with an existing entry.

    Returns (matched: bool, overlap: float, missing_signals: list).
    missing_signals lists which signals were unavailable (not failed).
    All available signals must match; missing signals are noted in the hint.
    """
    missing = []

    # Title
    new_words = title_words(new_title)
    existing_words = title_words(entry['title'])
    if not new_words or not existing_words:
        return False, 0.0, missing
    overlap = len(new_words & existing_words) / max(len(new_words), len(existing_words))
    if overlap < title_threshold:
        return False, overlap, missing

    # Author
    new_surnames = author_surnames(new_author)
    existing_surnames = author_surnames(entry['author'])
    if new_surnames and existing_surnames:
        if not (new_surnames & existing_surnames):
            return False, overlap, missing
    else:
        missing.append('author')

    # Year
    new_yr = (new_year or '').strip()
    existing_yr = (entry['year'] or '').strip()
    if new_yr and existing_yr:
        if new_yr != existing_yr:
            return False, overlap, missing
    else:
        missing.append('year')

    return True, overlap, missing


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--bib', required=True)
    parser.add_argument('--doi', default='')
    parser.add_argument('--title', default='')
    parser.add_argument('--author', default='')
    parser.add_argument('--year', default='')
    parser.add_argument('--new-pdf', default='')
    args = parser.parse_args()

    new_doi = normalize_doi(args.doi)
    new_title = args.title.lower().strip()
    new_author = args.author
    new_year = args.year.strip()
    new_pdf = args.new_pdf
    entries = load_bib(args.bib)

    new_arxiv_id, new_arxiv_ver = arxiv_id_and_version(new_pdf)

    for e in entries:
        citekey = e['citekey']
        existing_file = e['file']

        # --- Case 1: exact DOI match (deterministic) ---
        if new_doi and e['doi'] and new_doi == e['doi']:
            print(f"KEEP_EXISTING: citekey={citekey} existing_file={existing_file} "
                  f"reason=DOI match ({new_doi})")
            return

        # --- Case 2: arXiv ID match (deterministic version comparison) ---
        existing_arxiv_id, existing_arxiv_ver = arxiv_id_and_version(existing_file)
        if new_arxiv_id and existing_arxiv_id and new_arxiv_id == existing_arxiv_id:
            if new_arxiv_ver > existing_arxiv_ver:
                print(f"KEEP_NEW: citekey={citekey} existing_file={existing_file} "
                      f"reason=arXiv {new_arxiv_id} v{new_arxiv_ver} > v{existing_arxiv_ver}")
            else:
                print(f"KEEP_EXISTING: citekey={citekey} existing_file={existing_file} "
                      f"reason=arXiv {new_arxiv_id} existing v{existing_arxiv_ver} >= new v{new_arxiv_ver}")
            return

        # --- Case 3: new has DOI, existing is arXiv preprint (published beats preprint) ---
        if new_doi and existing_arxiv_id and not e['doi']:
            matched, overlap, missing = signals_match(
                new_title, new_author, new_year, e, title_threshold=0.6
            )
            if matched:
                caveat = f" (unconfirmed: {', '.join(missing)})" if missing else ""
                print(f"INFER: citekey={citekey} existing_file={existing_file} "
                      f"new_pdf={new_pdf} "
                      f"hint=New PDF has DOI {new_doi}, existing is arXiv preprint. "
                      f"Title overlap {overlap:.0%}, author+year match{caveat}. "
                      f"Read p1-2 of each: if same work, keep new (published) and delete arXiv file.")
                return

        # Different DOIs - definitely not a duplicate
        if new_doi and e['doi'] and new_doi != e['doi']:
            continue

        # --- Case 4: title + author + year similarity (all three required) ---
        if new_title and e['title']:
            matched, overlap, missing = signals_match(
                new_title, new_author, new_year, e, title_threshold=0.7
            )
            if matched:
                caveat = f" (unconfirmed: {', '.join(missing)})" if missing else ""
                print(f"INFER: citekey={citekey} existing_file={existing_file} "
                      f"new_pdf={new_pdf} "
                      f"hint=Title overlap {overlap:.0%}, author+year match{caveat}. "
                      f"No DOI confirmation - read p1-2 of each to verify same work.")
                return

    print("NO_MATCH")


if __name__ == '__main__':
    main()
#!/usr/bin/env python3
"""
Strip Anna's Archive suffixes and trailing UUIDs from PDF filenames in ~/org/pdfs/.

Patterns removed (from the stem, working right-to-left):
  " -- <anything> -- Anna's Archive"   (greedy, catches doi slugs before the UUID)
  " -- Anna's Archive"                 (standalone label)
  " -- <hex32>"                        (bare 32-char hex UUID with no label)

Dry-run by default. Pass --apply to actually rename.
"""
import re
import os
import glob
import argparse

pdfs_dir = os.path.expanduser("~/org/pdfs/")

ANNAS = r"Anna\u2019s Archive|Anna's Archive"  # U+2019 right single quote or ASCII

def clean_stem(stem: str) -> str:
    # Remove " -- <anything> -- Anna's Archive" (handles doi-slug + uuid before label)
    stem = re.sub(r"\s*--\s*.+--\s*(?:" + ANNAS + r")\s*$", "", stem, flags=re.IGNORECASE)
    # Remove standalone " -- Anna's Archive"
    stem = re.sub(r"\s*--\s*(?:" + ANNAS + r")\s*$", "", stem, flags=re.IGNORECASE)
    # Remove trailing bare UUID " -- <hex32>"
    stem = re.sub(r"\s*--\s*[0-9a-f]{32}\s*$", "", stem, flags=re.IGNORECASE)
    return stem.strip()

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--apply", action="store_true", help="Actually rename files (default: dry-run)")
    args = parser.parse_args()

    changes = []
    for path in sorted(glob.glob(os.path.join(pdfs_dir, "*.pdf"))):
        old_name = os.path.basename(path)
        stem, ext = os.path.splitext(old_name)
        new_stem = clean_stem(stem)
        new_name = new_stem + ext
        if new_name != old_name:
            changes.append((path, old_name, new_name))

    if not changes:
        print("Nothing to rename.")
    else:
        for path, old, new in changes:
            print(f"  {old}")
            print(f"→ {new}")
            print()
            if args.apply:
                new_path = os.path.join(pdfs_dir, new)
                if os.path.exists(new_path):
                    print(f"  SKIPPED (target already exists)")
                else:
                    os.rename(path, new_path)
        if not args.apply:
            print(f"{len(changes)} file(s) would be renamed. Pass --apply to rename.")
        else:
            print(f"{len(changes)} file(s) renamed.")
#!/usr/bin/env python3
"""
Find PDF files in ~/org/pdfs/ that have no matching file= entry in ~/org/bib/references.bib.

Output:
  MISSING:<comma-separated filenames>   (empty after colon if none)
  CITEKEYS:<comma-separated citekeys>
"""
import re
import os
import glob

bib_path = os.path.expanduser("~/org/bib/references.bib")
pdfs_dir = os.path.expanduser("~/org/pdfs/")

with open(bib_path) as f:
    bib = f.read()

file_vals = re.findall(r'file\s*=\s*\{([^}]+)\}', bib)
bib_files = {os.path.basename(v.strip()) for v in file_vals}

citekeys = set(re.findall(r'@\w+\{(\S+),', bib))

disk_files = {os.path.basename(p) for p in glob.glob(os.path.join(pdfs_dir, "*.pdf"))}

missing = sorted(disk_files - bib_files)

print("MISSING:" + ",".join(missing))
if missing:
    print("CITEKEYS:" + ",".join(sorted(citekeys)))
#!/usr/bin/env python3
"""Audit `abstract` fields in references.bib against the works' own abstracts.

Every abstract must be the work's own abstract, verbatim and complete, or be
absent. This script gathers reference texts, scores each bib abstract, and
proposes an action per entry: VERBATIM (keep), REPLACE (with a verified source
abstract), DELETE (the work has no abstract), or REVIEW (it cannot tell; check
the page by hand).

Usage (BIB is normally ~/org/bib/references.bib):
    audit_abstracts.py fetch    BIB [--keys k1,k2]   # fill the cache
    audit_abstracts.py score    BIB [--keys k1,k2]   # write the report, print results
    audit_abstracts.py show     BIB KEY              # bib text next to every source, for REVIEW
    audit_abstracts.py apply    BIB [--keys k1,k2] [--overrides FILE] [--dry-run]
    audit_abstracts.py preview  BIB FILE KEY         # text an override would write
    audit_abstracts.py reviewed BIB KEY NOTE         # record a hand-checked abstract

--keys limits fetch, score and apply to those citekeys (new entries); without
it the whole bib is audited.

Sources, cached under ~/.cache/citation-add/abstract-audit/:
    crossref  https://api.crossref.org/works/<doi>          (JATS converted)
    openalex  https://api.openalex.org/works/doi:<doi>      (inverted index rebuilt)
    arxiv     https://export.arxiv.org/api/query?id_list=   (for arXiv ids)
    search    CrossRef title lookup for entries without a DOI (exact title only;
              the bib's doi field is never changed)
    pdf       pdftotext on the `file` field (tesseract OCR for scans), plus
              regex cuts of the abstract

No personal data is sent: requests carry a generic User-Agent and no mailto.
Needs poppler (pdftotext, pdfinfo, pdftoppm) and tesseract.
"""

import difflib
import html
import json
import os
import re
import subprocess
import sys
import threading
import time
import unicodedata
import urllib.error
import urllib.parse
import urllib.request
import xml.etree.ElementTree as ET
from concurrent.futures import ThreadPoolExecutor

HERE = os.path.dirname(os.path.abspath(__file__))
STATE = os.path.expanduser("~/.cache/citation-add/abstract-audit")
CACHE = os.path.join(STATE, "cache")
REPORT = os.path.join(STATE, "report.json")
REVIEWED = os.path.join(STATE, "reviewed.json")
UA = "citation-add-abstract-audit/1.0 (python-urllib)"

# --------------------------------------------------------------------------
# BibTeX parsing with exact spans
# --------------------------------------------------------------------------


class Field:
    def __init__(self, name, value, vstart, vend, line_start, line_end, delim):
        self.name = name  # lowercased
        self.value = value  # raw text between delimiters
        self.vstart, self.vend = vstart, vend  # span of value (inside delimiters)
        self.line_start, self.line_end = line_start, line_end  # whole field incl. comma+newline
        self.delim = delim


class Entry:
    def __init__(self, etype, key, start, end):
        self.etype, self.key, self.start, self.end = etype, key, start, end
        self.fields = {}

    def get(self, name):
        f = self.fields.get(name)
        return f.value if f else None


def _match_brace(text, i):
    """text[i] == '{'; return index of the matching '}'."""
    depth = 0
    n = len(text)
    while i < n:
        c = text[i]
        if c == "\\":
            i += 2
            continue
        if c == "{":
            depth += 1
        elif c == "}":
            depth -= 1
            if depth == 0:
                return i
        i += 1
    raise ValueError("unbalanced braces")


def _delete_span(text, name_pos, after):
    """Span to cut so the field disappears and the layout around it stays intact.

    A field alone on its line(s) loses the whole line; a field sharing a line
    with others loses only itself, its comma and the spaces next to it.
    """
    ls = text.rfind("\n", 0, name_pos) + 1
    alone_before = text[ls:name_pos].strip() == ""
    cm = re.compile(r"[ \t]*,[ \t]*").match(text, after)
    end = cm.end() if cm else after
    at_eol = text[end : end + 1] == "\n"
    if alone_before and at_eol:
        return ls, end + 1
    if alone_before:
        return name_pos, end
    start = name_pos
    if at_eol:
        while start > 0 and text[start - 1] in " \t":
            start -= 1
    return start, end


def parse_bib(text):
    entries = []
    for m in re.finditer(r"(?m)^@(\w+)\s*\{", text):
        etype = m.group(1).lower()
        if etype in ("comment", "string", "preamble"):
            continue
        open_i = m.end() - 1
        close_i = _match_brace(text, open_i)
        km = re.compile(r"\s*([^,\s]+)\s*,").match(text, m.end())
        e = Entry(etype, km.group(1), m.start(), close_i + 1)
        i = km.end()
        while True:
            fm = re.compile(r"\s*([A-Za-z][\w-]*)\s*=\s*").match(text, i)
            if not fm or fm.end() > close_i:
                break
            name = fm.group(1).lower()
            j = fm.end()
            if text[j] == "{":
                vend = _match_brace(text, j)
                vstart, after, delim = j + 1, vend + 1, "{"
            elif text[j] == '"':
                k = j + 1
                depth = 0
                while True:
                    c = text[k]
                    if c == "\\":
                        k += 2
                        continue
                    if c == "{":
                        depth += 1
                    elif c == "}":
                        depth -= 1
                    elif c == '"' and depth == 0:
                        break
                    k += 1
                vstart, vend, after, delim = j + 1, k, k + 1, '"'
            else:
                bm = re.compile(r"[^,}\s]+").match(text, j)
                vstart, vend, after, delim = j, bm.end(), bm.end(), ""
            line_start, line_end = _delete_span(text, fm.start(1), after)
            e.fields[name] = Field(name, text[vstart:vend], vstart, vend, line_start, line_end, delim)
            i = after
            sm = re.compile(r"\s*,").match(text, i)
            if sm:
                i = sm.end()
        entries.append(e)
    return entries


# --------------------------------------------------------------------------
# Normalization and n-gram scoring
# --------------------------------------------------------------------------

LIGATURES = {"\ufb00": "ff", "\ufb01": "fi", "\ufb02": "fl", "\ufb03": "ffi", "\ufb04": "ffl", "\ufb05": "st", "\ufb06": "st"}
LATEX_ACCENT = re.compile(r"\\[`'^\"~=.uvHcdbtrk]\s*(?:\{\s*([A-Za-z])\s*\}|([A-Za-z]))")
LATEX_LETTERS = {r"\o": "o", r"\O": "O", r"\ss": "ss", r"\ae": "ae", r"\AE": "AE", r"\l": "l", r"\L": "L", r"\i": "i"}


def strip_latex(s):
    s = LATEX_ACCENT.sub(lambda m: m.group(1) or m.group(2), s)
    for k, v in LATEX_LETTERS.items():
        s = re.sub(re.escape(k) + r"(?![A-Za-z])", v, s)
    s = re.sub(r"\\(?:textit|textbf|emph|textsc|texttt|textrm|mathrm|url)\s*\{", "{", s)
    s = re.sub(r"\\([&%#$_{}])", r"\1", s)
    s = s.replace("{", "").replace("}", "")
    s = s.replace("~", " ").replace("--", "-")
    return s


def normalize(s):
    """Lowercased word tokens, robust to quoting, dashes, hyphenation, accents, LaTeX."""
    s = strip_latex(s)
    for k, v in LIGATURES.items():
        s = s.replace(k, v)
    s = s.replace("\u00ad", "")  # soft hyphen
    s = re.sub(r"[\u2018\u2019\u201a\u201b\u2032`]", "'", s)
    s = re.sub(r"[\u201c\u201d\u201e\u201f\u2033]", '"', s)
    s = re.sub(r"[\u2010\u2011\u2012\u2013\u2014\u2015\u2212]", "-", s)
    # rejoin words broken across lines, and treat in-word hyphens as joins,
    # so "exam-\nple", "data-\ndriven" and "data-driven" all compare equal
    s = re.sub(r"(\w)-\s*\n\s*(\w)", r"\1\2", s)
    s = re.sub(r"(\w)-(\w)", r"\1\2", s)
    s = re.sub(r"(\w)'(\w)", r"\1\2", s)
    s = unicodedata.normalize("NFKD", s)
    s = "".join(c for c in s if not unicodedata.combining(c))
    s = s.lower()
    s = re.sub(r"\s+", " ", s)
    return re.findall(r"[a-z0-9]+", s)


def grams(tokens, n=5):
    if not tokens:
        return set()
    if len(tokens) < n:
        return {tuple(tokens)}
    return {tuple(tokens[i : i + n]) for i in range(len(tokens) - n + 1)}


def containment(a_grams, b_grams):
    """Fraction of a's n-grams found in b."""
    if not a_grams:
        return 0.0
    return len(a_grams & b_grams) / len(a_grams)


def align_ratio(a, b, n=20):
    """Similarity of a to the stretch of b it most plausibly came from.

    Both sides are reduced to normalized letters with spacing removed, so PDF
    artifacts ("whic hthe", a dropped "fi" ligature) cost a few letters, while
    words or sentences dropped from the middle make the window longer than a
    and pull the ratio down. Returns 0.0 when no window is found.
    """
    A, B = "".join(normalize(a)), "".join(normalize(b))
    if len(A) < n or len(B) < n:
        return 0.0
    first = last = None
    for i in range(len(A) - n + 1):
        k = B.find(A[i : i + n])
        if k >= 0:
            first = max(0, k - i)
            break
    if first is None:
        return 0.0
    for j in range(len(A) - n, -1, -1):
        k = B.find(A[j : j + n], first)
        if k >= 0:
            last = k + n + (len(A) - j - n)
            break
    if last is None or last <= first or last - first > 3 * len(A):
        return 0.0
    W = B[first:last]
    sm = difflib.SequenceMatcher(None, A, W, autojunk=False)
    matched = sum(b.size for b in sm.get_matching_blocks() if b.size >= 4)
    return matched / max(len(A), len(W))


def title_sim(a, b):
    ta, tb = set(normalize(a or "")), set(normalize(b or ""))
    if not ta or not tb:
        return 0.0
    return len(ta & tb) / min(len(ta), len(tb))


# --------------------------------------------------------------------------
# Fetching with cache and per-host rate limits
# --------------------------------------------------------------------------


class RateLimiter:
    def __init__(self, interval):
        self.interval, self.lock, self.last = interval, threading.Lock(), 0.0

    def wait(self):
        with self.lock:
            delay = self.last + self.interval - time.monotonic()
            if delay > 0:
                time.sleep(delay)
            self.last = time.monotonic()


LIMITS = {"crossref": RateLimiter(0.5), "openalex": RateLimiter(0.25), "arxiv": RateLimiter(3.5)}


def http_get(url, limiter, retries=3):
    for attempt in range(retries):
        limiter.wait()
        req = urllib.request.Request(url, headers={"User-Agent": UA})
        try:
            with urllib.request.urlopen(req, timeout=30) as r:
                return r.status, r.read().decode("utf-8", "replace")
        except urllib.error.HTTPError as e:
            if e.code in (404, 410):
                return e.code, ""
            if e.code in (429, 500, 502, 503, 504) and attempt < retries - 1:
                time.sleep(5 * (attempt + 1))
                continue
            return e.code, ""
        except Exception as e:  # network errors
            if attempt < retries - 1:
                time.sleep(5 * (attempt + 1))
                continue
            return -1, str(e)
    return -1, ""


def cache_path(kind, key):
    safe = re.sub(r"[^A-Za-z0-9._-]+", "_", key)
    d = os.path.join(CACHE, kind)
    os.makedirs(d, exist_ok=True)
    return os.path.join(d, safe + ".json")


def cached(kind, key, fn):
    p = cache_path(kind, key)
    if os.path.exists(p):
        with open(p, encoding="utf-8") as f:
            return json.load(f)
    val = fn()
    if val.get("status", 0) in (-1, 429) or (isinstance(val.get("status"), int) and val["status"] >= 500):
        return val  # do not cache transient failures
    with open(p, "w", encoding="utf-8") as f:
        json.dump(val, f, ensure_ascii=False, indent=1)
    return val


def jats_to_text(s):
    if not s:
        return ""
    # drop a leading "Abstract"/"Summary" heading; keep other section titles as words
    s = re.sub(r"<jats:title>\s*(abstract|summary|a b s t r a c t)\s*</jats:title>", " ", s, flags=re.I)
    s = re.sub(r"<jats:title>\s*([^<]*?)\s*</jats:title>", lambda m: " " + m.group(1).rstrip(":") + ": ", s)
    s = re.sub(r"</?jats:(p|sec)[^>]*>", " ", s)
    s = re.sub(r"<[^>]+>", "", s)
    s = html.unescape(s)
    s = re.sub(r"\s+", " ", s).strip()
    s = re.sub(r"^(abstract|summary)\s*[:.\-\u2014]?\s+", "", s, flags=re.I)
    return s


def fetch_crossref(doi):
    def go():
        url = "https://api.crossref.org/works/" + urllib.parse.quote(doi, safe="/:;()")
        status, body = http_get(url, LIMITS["crossref"])
        out = {"status": status}
        if status == 200:
            w = json.loads(body).get("message", {})
            out.update(
                title=(w.get("title") or [""])[0],
                type=w.get("type", ""),
                abstract_raw=w.get("abstract", ""),
            )
        return out

    r = cached("crossref", doi.lower(), go)
    if r.get("abstract_raw"):
        r["abstract"] = strip_boilerplate(jats_to_text(r["abstract_raw"]))
    return r


def fetch_openalex(doi):
    def go():
        url = (
            "https://api.openalex.org/works/doi:"
            + urllib.parse.quote(doi, safe="/:;()")
            + "?select=id,title,type,publication_year,abstract_inverted_index"
        )
        status, body = http_get(url, LIMITS["openalex"])
        out = {"status": status}
        if status == 200:
            w = json.loads(body)
            out.update(id=w.get("id"), title=w.get("title") or "", type=w.get("type", ""), inv=w.get("abstract_inverted_index"))
        return out

    r = cached("openalex", doi.lower(), go)
    inv = r.get("inv")
    if inv:
        pos = {}
        for word, idxs in inv.items():
            for i in idxs:
                pos[i] = word
        r["abstract"] = strip_boilerplate(re.sub(r"\s+", " ", " ".join(pos[i] for i in sorted(pos))).strip())
    return r


def fetch_arxiv(ids):
    """Fetch many arXiv ids; returns {id: {...}} and caches each id."""
    todo = [i for i in ids if not os.path.exists(cache_path("arxiv", i))]
    for k in range(0, len(todo), 50):
        batch = todo[k : k + 50]
        url = "https://export.arxiv.org/api/query?max_results=100&id_list=" + ",".join(batch)
        status, body = http_get(url, LIMITS["arxiv"])
        if status != 200:
            print(f"arxiv batch failed: {status}", file=sys.stderr)
            continue
        ns = {"a": "http://www.w3.org/2005/Atom"}
        root = ET.fromstring(body)
        found = {}
        for ent in root.findall("a:entry", ns):
            eid = ent.findtext("a:id", "", ns)
            m = re.search(r"abs/([^v\s]+?)(v\d+)?$", eid)
            if not m:
                continue
            found[m.group(1)] = {
                "status": 200,
                "title": re.sub(r"\s+", " ", ent.findtext("a:title", "", ns)).strip(),
                "abstract": re.sub(r"\s+", " ", ent.findtext("a:summary", "", ns)).strip(),
            }
        for i in batch:
            val = found.get(i, {"status": 404})
            with open(cache_path("arxiv", i), "w", encoding="utf-8") as f:
                json.dump(val, f, ensure_ascii=False, indent=1)
    out = {}
    for i in ids:
        with open(cache_path("arxiv", i), encoding="utf-8") as f:
            out[i] = json.load(f)
    return out


def pdftotext(path, first=None, last=None):
    cmd = ["pdftotext", "-enc", "UTF-8"]
    if first:
        cmd += ["-f", str(first)]
    if last:
        cmd += ["-l", str(last)]
    cmd += [path, "-"]
    try:
        r = subprocess.run(cmd, capture_output=True, timeout=120)
        return r.stdout.decode("utf-8", "replace")
    except Exception:
        return ""


def pdf_info(path):
    try:
        r = subprocess.run(["pdfinfo", path], capture_output=True, timeout=30)
        m = re.search(rb"Pages:\s+(\d+)", r.stdout)
        return int(m.group(1)) if m else 0
    except Exception:
        return 0


sys.path.insert(0, HERE)
try:
    from extract_abstract import try_patterns as _skill_patterns  # noqa: E402
except Exception:  # pragma: no cover
    _skill_patterns = None

CUT_START = re.compile(
    r"(?:^|\n)[ \t\f]*(?:a\s?b\s?s\s?t\s?r\s?a\s?c\s?t|summary)\b[ \t]*[:.\u2014\u2013-]*[ \t]*", re.I
)
CUT_END = re.compile(
    r"\n\s*(?:key\s?words?|index\s+terms|ccs\s+concepts|categories\s+and\s+subject|general\s+terms|"
    r"additional\s+key\s+words|acm\s+reference|(?:1|i)\.?\s+introduction|introduction\s*\n|1\s*\n|1\.?\s+[A-Z])",
    re.I,
)


def pdf_cuts(text):
    """Candidate regex cuts of the abstract, best first: the text after an
    "Abstract" heading, ending at the next heading (Keywords, Index Terms,
    1 Introduction) when that is the same block, else at the first blank line or
    page break, else at a blank line across a page break, else at the heading.
    The add-bibs skill's pattern library comes last. Returns [(text, how)]."""
    out = []
    words = lambda t: len(re.findall(r"\w+", t or ""))
    for m in CUT_START.finditer(text):
        rest = text[m.end() :].lstrip()
        he = CUT_END.search(rest)
        head_cut = rest[: he.start()] if he else None
        pb = re.search(r"\n[ \t\f]*\n|\f", rest)
        para_cut = rest[: pb.start()] if pb else None
        pb2 = re.search(r"\n[ \t]*\n", rest)
        para2_cut = rest[: pb2.start()] if pb2 else None
        if head_cut and para_cut and len(head_cut) <= 1.15 * len(para_cut) and words(head_cut) >= 40:
            out.append((head_cut, "heading"))
        if para_cut and words(para_cut) >= 40:
            after = rest[pb.end() :].lstrip()
            nb = re.search(r"\n[ \t\f]*\n|\f", after)
            nxt = after[: nb.start()] if nb else after
            prose = words(nxt) >= 30 and not CUT_END.match("\n" + nxt) and re.search(r"[.?!]\s*$", nxt.strip())
            out.append((para_cut, "paragraph+prose" if prose else "paragraph"))
        if para2_cut and para2_cut != para_cut and words(para2_cut) >= 40:
            out.append((para2_cut, "paragraph-across-page"))
        if head_cut and 40 <= words(head_cut) <= 700:
            out.append((head_cut, "heading-long"))
    if _skill_patterns:
        r = _skill_patterns(text)
        if r:
            out.append((r[0], "skill:" + r[1]))
    seen, uniq = set(), []
    for t, how in out:
        t = t.strip()
        if t not in seen:
            seen.add(t)
            uniq.append((t, how))
    return uniq


def ocr_pdf(path, first=None, last=None):
    """OCR a scanned PDF with pdftoppm + tesseract; pages joined with form feeds."""
    import tempfile

    out = []
    with tempfile.TemporaryDirectory() as d:
        cmd = ["pdftoppm", "-r", "300", "-gray", "-png"]
        if first:
            cmd += ["-f", str(first)]
        if last:
            cmd += ["-l", str(last)]
        subprocess.run(cmd + [path, os.path.join(d, "p")], capture_output=True, timeout=900)
        for img in sorted(os.listdir(d)):
            r = subprocess.run(["tesseract", os.path.join(d, img), "-", "--psm", "1"], capture_output=True, timeout=300)
            out.append(r.stdout.decode("utf-8", "replace"))
    return "\f".join(out)


def fetch_pdf(key, path):
    def go():
        p = os.path.expanduser(path)
        if not os.path.exists(p):
            return {"status": 404, "path": path}
        full = pdftotext(p)
        head, wide, ocr = pdftotext(p, 1, 3), pdftotext(p, 1, 15), False
        if not usable_text(full):
            full, ocr = ocr_pdf(p), True
            pages = full.split("\f")
            head, wide = "\f".join(pages[:3]), "\f".join(pages[:15])
        return {"status": 200, "path": path, "pages": pdf_info(p), "ocr": ocr, "full": full, "head": head, "wide": wide}

    return cached("pdf", key, go)


def search_doi(e):
    """Find a DOI for an entry that has none: CrossRef bibliographic query, accepted
    only on an exact normalized title match (or main title before a colon) and a
    year within one. The bib's own fields are never changed by this."""
    title = strip_latex(e.get("title") or "")
    author = strip_latex((e.get("author") or e.get("editor") or "").split(" and ")[0].split(",")[0])
    year = re.sub(r"\D", "", e.get("year") or "")

    def go():
        q = {"query.bibliographic": title, "rows": "5", "select": "DOI,title,issued,type"}
        if author:
            q["query.author"] = author
        url = "https://api.crossref.org/works?" + urllib.parse.urlencode(q)
        status, body = http_get(url, LIMITS["crossref"])
        out = {"status": status, "items": []}
        if status == 200:
            for it in json.loads(body).get("message", {}).get("items", []):
                y = ((it.get("issued") or {}).get("date-parts") or [[None]])[0][0]
                out["items"].append({"doi": it.get("DOI"), "title": (it.get("title") or [""])[0], "year": y, "type": it.get("type")})
        return out

    r = cached("search", e.key, go)
    want = normalize(title)
    want_main = normalize(title.split(":")[0])
    for it in r.get("items", []):
        got = normalize(it["title"])
        if got != want and got != want_main and normalize(it["title"].split(":")[0]) != want:
            continue
        if year and it.get("year") and abs(int(it["year"]) - int(year[:4])) > 1:
            continue
        return it["doi"]
    return None


# --------------------------------------------------------------------------
# Per-entry source gathering
# --------------------------------------------------------------------------

ARXIV_RE = re.compile(r"(?:arxiv[.:/ ]+(?:abs/|pdf/)?|10\.48550/arxiv\.)(\d{4}\.\d{4,5}|[a-z-]+(?:\.[A-Z]{2})?/\d{7})", re.I)


def arxiv_id(e):
    doi = (e.get("doi") or "").strip()
    m = re.match(r"10\.48550/arxiv\.(.+)$", doi, re.I)
    if m:
        return re.sub(r"v\d+$", "", m.group(1))
    if (e.get("eprinttype") or "").lower() == "arxiv" and e.get("eprint"):
        return re.sub(r"v\d+$", "", e.get("eprint").strip())
    for f in ("url", "journal", "note", "howpublished"):
        v = e.get(f) or ""
        m = ARXIV_RE.search(v)
        if m:
            return re.sub(r"v\d+$", "", m.group(1))
    return None


def load(bib):
    with open(bib, encoding="utf-8") as f:
        text = f.read()
    return text, parse_bib(text)


def entry_doi(e, search=False):
    """(doi, how): the bib's own DOI, else one found by title search (cached),
    else None. arXiv DOIs are handled by the arXiv source instead."""
    doi = (e.get("doi") or "").strip()
    if doi:
        return (None, None) if re.match(r"10\.48550/", doi, re.I) else (doi, "bib")
    if arxiv_id(e):
        return None, None
    if search or os.path.exists(cache_path("search", e.key)):
        found = search_doi(e)
        if found:
            return found, "title-search"
    return None, None


def cmd_fetch(bib, keys=None):
    text, entries = load(bib)
    with_abs = [e for e in entries if "abstract" in e.fields and (not keys or e.key in keys)]
    dois = sorted({d for e in with_abs if (d := entry_doi(e, search=True)[0])})
    aids = sorted({a for e in with_abs if (a := arxiv_id(e))})
    print(f"{len(with_abs)} entries with abstracts, {len(dois)} DOIs, {len(aids)} arXiv ids", file=sys.stderr)

    def pdf_job(e):
        if e.get("file"):
            fetch_pdf(e.key, e.get("file"))

    with ThreadPoolExecutor(6) as ex:
        list(ex.map(pdf_job, with_abs))
    print("pdfs done", file=sys.stderr)

    def api_job(kind):
        fn = fetch_crossref if kind == "crossref" else fetch_openalex
        for i, d in enumerate(dois):
            r = fn(d)
            if r.get("status") not in (200, 404):
                print(f"{kind} {d}: {r.get('status')}", file=sys.stderr)
        print(f"{kind} done", file=sys.stderr)

    threads = [threading.Thread(target=api_job, args=(k,)) for k in ("crossref", "openalex")]
    threads.append(threading.Thread(target=fetch_arxiv, args=(aids,)))
    for t in threads:
        t.start()
    for t in threads:
        t.join()


# --------------------------------------------------------------------------
# Scoring and classification
# --------------------------------------------------------------------------

T_HI = 0.90  # bib abstract found in a reference text: verbatim
T_LO = 0.30  # below this against every source: clear miss
T_REV = 0.85  # reference abstract covered by bib abstract: complete
T_TITLE = 0.6  # API title vs bib title
T_AGREE = 0.80  # API abstract found in PDF full text (when PDF text is usable)
T_GAP = 6  # a verified abstract may not have a run this long missing from a "verbatim" bib
T_ALIGN = 0.97  # letter alignment with the PDF window: verbatim despite extraction damage

END_WORDS = {
    "keywords", "keyword", "key", "index", "ccs", "categories", "general", "additional", "acm",
    "introduction", "1", "i", "permission", "copyright", "c", "author", "authors", "received",
    "citation", "doi", "http", "https", "www", "corresponding", "email", "e", "references",
    "subject", "jel", "contents", "table", "preprint", "submitted", "published", "proceedings",
    "background", "motivation", "overview", "a", "1introduction", "th", "rd", "nd", "st",
}


def usable_text(t):
    if not t:
        return False
    toks = re.findall(r"[A-Za-z]{2,}", t)
    return len(toks) > 300


_DICT = None


def dict_word(w):
    """True if w (or w minus a common suffix) is in the system word list."""
    global _DICT
    if _DICT is None:
        try:
            with open("/usr/share/dict/words") as f:
                _DICT = {x.strip().lower() for x in f}
        except OSError:
            _DICT = set()
    w = w.lower()
    if w in _DICT:
        return True
    for suf in ("s", "es", "ed", "d", "ing", "ly", "er", "ers", "ies", "al", "ity", "ment", "ments", "ness"):
        if w.endswith(suf) and len(w) - len(suf) >= 3:
            stem = w[: -len(suf)]
            if stem in _DICT or stem + "e" in _DICT or stem + "y" in _DICT:
                return True
    return False


def clean_pdf_cut(cut, full):
    """Turn a pdftotext abstract cut into a plain paragraph.

    Line-broken hyphens are rejoined only when the joined word occurs elsewhere
    in the PDF text and the hyphenated form does not; otherwise the hyphen stays.
    """
    if not cut:
        return None
    for k, v in LIGATURES.items():
        cut = cut.replace(k, v)
    words_full = full.lower()

    def join(m):
        a, b = m.group(1), m.group(2)
        joined, hyph = (a + b).lower(), (a + "-" + b).lower()
        jw = len(re.findall(r"\b" + re.escape(joined) + r"\b", words_full))
        hw = len(re.findall(r"\b" + re.escape(hyph) + r"\b", words_full))
        if hw > 0 and jw == 0:
            return a + "-" + b
        if jw > 0:
            return a + b
        if dict_word(joined):
            return a + b
        if dict_word(a) and dict_word(b):
            return a + "-" + b
        return a + b if b[:1].islower() else a + "-" + b

    cut = re.sub(r"(\w+)-\s*\n\s*(\w+)", join, cut)
    cut = cut.replace("\u00ad", "")
    cut = re.sub(r"[-\u2010\u2011]{2,}", "-", cut)
    cut = cut.replace("\u2018\u2018", "\u201c").replace("\u2019\u2019", "\u201d")
    cut = re.sub(r"\s+", " ", cut).strip()

    def split_joined(m):
        # "contextaware" -> "context-aware" when the PDF itself spells it that way elsewhere
        w = m.group(0)
        if dict_word(w):
            return w
        for i in range(2, len(w) - 1):
            a, b = w[:i], w[i:]
            if re.search(r"\b" + re.escape(a + "-" + b) + r"\b", full, re.I):
                return a + "-" + b
        return w

    cut = re.sub(r"\b[a-z]{5,}\b", split_joined, cut)
    return strip_boilerplate(cut)


BOILER_TAIL = re.compile(
    r"\s*(?:\u00a9|\(c\)\s*\d{4}|\u00d3\s*\d{4}|[\x02\x03]\s*\d{4}|Copyright\b|Received:?\s+(?:\d|[A-Z][a-z]+ \d)"
    r"|Article history|All rights reserved).*$",
    re.S,
)


def strip_boilerplate(t):
    """Drop a trailing publisher copyright or received/accepted block."""
    m = BOILER_TAIL.search(t)
    if m and m.start() > 0.6 * len(t) and len(re.findall(r"\w+", t[: m.start()])) >= 40:
        return t[: m.start()].rstrip()
    return t


JUNK = re.compile(
    r"requests? for reprints|this (?:research|work|study) was (?:supported|funded)|correspondence (?:concerning|should)"
    r"|\u00a9|\(c\) \d{4}|copyright|all rights reserved|published by elsevier|key\s?words?\s*[:\u2014-]|https?://doi\.org"
    r"|e-?mail\s*:|received \w+ \d{1,2},? \d{4}|graphical abstract|\[\d+(?:\s*[,\u2013-]\s*\d+)*\]",
    re.I,
)


def hyphen_artifacts(t, full):
    """Hyphenated words in t that the PDF text shows only as a line-break split."""
    out = []
    for m in re.finditer(r"\b([a-z]+)-([a-z]+)\b", t):
        a, b = m.group(1), m.group(2)
        if re.search(re.escape(a) + r"-\s*\n\s*" + re.escape(b) + r"\b", full) and not re.search(
            r"\b" + re.escape(a) + "-" + re.escape(b) + r"\b", full
        ):
            out.append(m.group(0))
    return out


def ends_cleanly(t):
    t = t.rstrip()
    return bool(re.search(r"[.?!)\]\"'\u201d\u2019]$", t)) and not re.search(r"(\.\.\.|\u2026)$", t)


def trustworthy_api(abs_text):
    if not abs_text:
        return False
    n = len(normalize(abs_text))
    return 40 <= n <= 1200 and not JUNK.search(abs_text)


def trustworthy_pdf_cut(cut):
    if not cut:
        return False
    toks = normalize(cut)
    if not (60 <= len(toks) <= 700):
        return False
    if not re.match(r"^[A-Z0-9\"'\u201c(]", cut):
        return False
    if not re.search(r"(?:[.?!)\]\"\u201d\u2019]|https?://\S+|\w\.\w+/\S+)$", cut):
        return False
    if re.match(r"(?i)key\s?words?\b|index terms\b", cut):
        return False
    bad = (
        r"@|\u00a9|copyright|permission to make|all rights reserved|e-?mail\b"
        r"|\b(?:department|dept\.?|school|faculty|institute) of [A-Z]"
        r"|\[\d+(?:\s*[,\u2013-]\s*\d+)*\]|\bhistory:|supplemental material:|funding:|licensed under"
    )
    if re.search(bad, cut, re.I):
        return False
    return True


def gaps(a, ref, n=5):
    """Runs of ref's tokens not covered by any n-gram shared with a.
    Returns [(start, length, 'start'|'mid'|'end', preview)]."""
    rt, ag = normalize(ref), grams(normalize(a), n)
    cov = [False] * len(rt)
    for i in range(len(rt) - n + 1):
        if tuple(rt[i : i + n]) in ag:
            for j in range(i, i + n):
                cov[j] = True
    runs, i = [], 0
    while i < len(rt):
        if cov[i]:
            i += 1
            continue
        j = i
        while j < len(rt) and not cov[j]:
            j += 1
        where = "start" if i == 0 else ("end" if j == len(rt) else "mid")
        runs.append((i, j - i, where, " ".join(rt[i:j][:10])))
        i = j
    return runs


def continuation_ok(bib_toks, full_toks):
    """True if, in the PDF text, the bib abstract is followed by a heading-like token
    (no truncation), or if we cannot locate it."""
    if len(bib_toks) < 10 or not full_toks:
        return None
    tail = tuple(bib_toks[-5:])
    n = len(full_toks)
    hits = [i for i in range(n - 4) if tuple(full_toks[i : i + 5]) == tail]
    if not hits:
        return None
    head = tuple(bib_toks[:5])
    starts = [i for i in range(n - 4) if tuple(full_toks[i : i + 5]) == head]
    if starts:
        hits = [h for h in hits if h >= starts[0]] or hits
    i = hits[0] + 5
    nxt = full_toks[i : i + 3]
    return bool(nxt) and nxt[0] in END_WORDS


def score_entry(e):
    bib_abs = e.get("abstract")
    bib_toks = normalize(bib_abs)
    bg = grams(bib_toks)
    rec = {
        "key": e.key,
        "type": e.etype,
        "doi": e.get("doi") or "",
        "file": e.get("file") or "",
        "bib_words": len(bib_toks),
        "bib_head": re.sub(r"\s+", " ", bib_abs)[:120],
        "sources": {},
        "flags": [],
    }
    refs = {}  # name -> abstract text
    full_text = None

    doi, doi_how = entry_doi(e)
    if doi_how == "title-search":
        rec["doi_found"] = doi
    if doi:
        for kind, fn in (("crossref", fetch_crossref), ("openalex", fetch_openalex)):
            p = cache_path(kind, doi.lower())
            if not os.path.exists(p):
                rec["sources"][kind] = {"status": "uncached"}
                continue
            r = fn(doi)
            s = {"status": r.get("status")}
            if r.get("title"):
                s["title_sim"] = round(title_sim(r["title"], e.get("title")), 2)
            if r.get("abstract"):
                s["words"] = len(normalize(r["abstract"]))
                refs[kind] = (r["abstract"], s)
            rec["sources"][kind] = s
    aid = arxiv_id(e)
    if aid:
        p = cache_path("arxiv", aid)
        if os.path.exists(p):
            r = json.load(open(p, encoding="utf-8"))
            s = {"status": r.get("status"), "id": aid}
            if r.get("title"):
                s["title_sim"] = round(title_sim(r["title"], e.get("title")), 2)
            if r.get("abstract"):
                s["words"] = len(normalize(r["abstract"]))
                refs["arxiv"] = (r["abstract"], s)
            rec["sources"]["arxiv"] = s
    if e.get("file"):
        p = cache_path("pdf", e.key)
        if os.path.exists(p):
            r = json.load(open(p, encoding="utf-8"))
            s = {"status": r.get("status"), "pages": r.get("pages")}
            if r.get("status") == 200:
                full_text = r.get("full") or ""
                s["usable"] = usable_text(full_text)
                s["chars"] = len(full_text)
                s["title_in_head"] = round(
                    containment(set(normalize(e.get("title") or "")), set(normalize((r.get("head") or "")[:4000]))), 2
                )
                s["ocr"] = r.get("ocr", False)
                cands = [(t, h) for t, h in pdf_cuts(r.get("head") or "")]
                cands += [(t, h + "(wide)") for t, h in pdf_cuts(r.get("wide") or "")]
                cands = [(clean_pdf_cut(t, full_text), h) for t, h in cands]
                good = [c for c in cands if trustworthy_pdf_cut(c[0]) and "+prose" not in c[1]]
                if good or cands:
                    cut, how = (good or cands)[0]
                    s["cut_how"] = how
                    s["cut_words"] = len(normalize(cut))
                    s["cut_trust"] = trustworthy_pdf_cut(cut)
                    refs["pdf_cut"] = (cut, s)
                s["has_abstract_word"] = bool(
                    re.search(r"(?im)^[ \t\f]*(?:a\s?b\s?s\s?t\s?r\s?a\s?c\s?t|summary)\b", (r.get("wide") or ""))
                )
            rec["sources"]["pdf"] = s

    full_toks = normalize(full_text) if full_text else []
    full_g = grams(full_toks)
    pdf_usable = bool(full_text) and usable_text(full_text)

    # forward scores: bib abstract inside each reference
    fwd = {}
    for name, (txt, s) in refs.items():
        if name == "pdf_cut":
            continue
        fwd[name] = containment(bg, grams(normalize(txt)))
    if full_text:
        fwd["pdf_full"] = containment(bg, full_g)
    if "pdf_cut" in refs:
        fwd["pdf_cut"] = containment(bg, grams(normalize(refs["pdf_cut"][0])))
    rec["fwd"] = {k: round(v, 3) for k, v in fwd.items()}

    # reverse scores: each reference abstract covered by the bib abstract (completeness)
    rev = {}
    agree = {}
    for name, (txt, s) in refs.items():
        rg = grams(normalize(txt))
        rev[name] = containment(rg, bg)
        if pdf_usable and name != "pdf_cut":
            agree[name] = containment(rg, full_g)
    rec["rev"] = {k: round(v, 3) for k, v in rev.items()}
    rec["api_in_pdf"] = {k: round(v, 3) for k, v in agree.items()}

    # alignment against the PDF text tolerates extraction damage but not edits
    align = align_ratio(bib_abs, full_text) if pdf_usable else 0.0
    rec["align_pdf"] = round(align, 3)

    best = max(fwd.values()) if fwd else 0.0
    rec["best"] = round(best, 3)
    rec["best_src"] = max(fwd, key=fwd.get) if fwd else ""

    pdf_s = rec["sources"].get("pdf", {})
    cut_trusted = "pdf_cut" in refs and pdf_s.get("cut_trust") and pdf_s.get("title_in_head", 0) >= 0.8

    def api_ok(name):
        """An API abstract is used only if it is the right work, clean, and complete."""
        txt, s = refs[name]
        problems = []
        if s.get("title_sim", 1.0) < T_TITLE:
            problems.append("title")
        if not trustworthy_api(txt):
            problems.append("junk-or-length")
        if full_text:
            arts = hyphen_artifacts(txt, full_text)
            if arts:
                problems.append("hyphenation:" + ",".join(arts[:3]))
        if pdf_usable:
            if agree.get(name, 0) < T_AGREE:
                problems.append(f"not-in-pdf:{agree.get(name, 0):.2f}")
            xg = grams(normalize(txt))
            if cut_trusted:
                cg = grams(normalize(refs["pdf_cut"][0]))
                x_in_cut, cut_in_x = containment(xg, cg), containment(cg, xg)
                if x_in_cut < T_REV:
                    problems.append(f"beyond-pdf-abstract:{x_in_cut:.2f}")
                complete = cut_in_x >= T_REV or continuation_ok(normalize(txt), full_toks)
            else:
                complete = ends_cleanly(txt)
            if not complete:
                problems.append("maybe-truncated")
        elif not ends_cleanly(txt):
            problems.append("maybe-truncated")
        s["problems"] = problems
        return not problems

    api_pref = [n for n in ("crossref", "arxiv", "openalex") if n in refs]
    good_api = [n for n in api_pref if api_ok(n)]
    for n in api_pref:
        if n not in good_api:
            rec["flags"].append(f"untrusted:{n}")

    # the replacement we would use: trusted API abstract first, then a clean PDF cut
    cand = None
    if good_api:
        cand = good_api[0]
    elif cut_trusted and pdf_usable:
        cand = "pdf_cut"
    if cand:
        rec["candidate_src"], rec["candidate"] = cand, refs[cand][0]

    verbatim_api = [n for n in api_pref if fwd[n] >= T_HI]
    verbatim_pdf = pdf_usable and (fwd.get("pdf_full", 0) >= T_HI or align >= T_ALIGN)

    action, why, use = None, "", None
    if verbatim_api:
        comp = max(rev[n] for n in verbatim_api)
        trusted = [n for n in good_api if n in verbatim_api]
        if trusted:
            # complete means no run of T_GAP+ words of a verified abstract is missing
            missing = {n: [g for g in gaps(bib_abs, refs[n][0]) if g[1] >= T_GAP] for n in trusted}
            rec["missing"] = {n: [f"{g[2]}:{g[3]}" for g in v] for n, v in missing.items() if v}
            whole = [n for n in trusted if not missing[n]]
            cut_whole = (
                cut_trusted
                and fwd.get("pdf_cut", 0) >= T_HI
                and not [g for g in gaps(bib_abs, refs["pdf_cut"][0]) if g[1] >= T_GAP and g[2] != "start"]
            )
            if whole:
                action, why = "VERBATIM", f"in {','.join(verbatim_api)}; nothing missing from {whole[0]}"
            elif cut_whole:
                action, why = "VERBATIM", f"nothing missing from the PDF abstract; {trusted[0]} has another version"
            else:
                n = trusted[0]
                where = ",".join(sorted({g[2] for g in missing[n]}))
                action, why, use = "REPLACE", f"incomplete: {where} of {n} missing", n
        elif comp >= T_REV:
            action, why = "VERBATIM", f"in untrusted {','.join(verbatim_api)}; covers {comp:.2f} of it"
        else:
            action, why = "REVIEW", f"in untrusted {','.join(verbatim_api)}; covers {comp:.2f} of it"
    elif verbatim_pdf:
        cut_ok = (
            "pdf_cut" in refs
            and fwd.get("pdf_cut", 0) >= T_HI
            and not [g for g in gaps(bib_abs, refs["pdf_cut"][0]) if g[1] >= T_GAP and g[2] != "start"]
        )
        cont = continuation_ok(bib_toks, full_toks)
        rec["continues_to_heading"] = cont
        differs = f"; {good_api[0]} has another version" if good_api else ""
        # a cut-short copy: the bib text sits inside a clean PDF abstract that goes on
        cut_missing = []
        if cut_trusted and fwd.get("pdf_cut", 0) >= T_HI:
            cut_missing = [g for g in gaps(bib_abs, refs["pdf_cut"][0]) if g[1] >= T_GAP and g[2] != "start"]
        if cut_ok:
            action, why = "VERBATIM", "in PDF; covers the PDF abstract cut" + differs
        elif cont:
            action, why = "VERBATIM", "in PDF; followed by a heading" + differs
        elif cut_missing:
            where = ",".join(sorted({g[2] for g in cut_missing}))
            action, why, use = "REPLACE", f"incomplete: {where} of the PDF abstract missing", "pdf_cut"
        else:
            action, why = "REVIEW", "in PDF; completeness unclear" + differs
    elif best < T_LO:
        if cand:
            action, why, use = "REPLACE", f"paraphrase; {cand} abstract", cand
        elif api_pref:
            action, why = "REVIEW", "paraphrase; API abstract untrusted"
        elif pdf_usable and not pdf_s.get("has_abstract_word"):
            if pdf_s.get("title_in_head", 0) >= 0.5:
                action, why = "DELETE", "paraphrase; source has no abstract"
            else:
                action, why = "REVIEW", "paraphrase; PDF title mismatch"
        elif pdf_usable:
            action, why = "REVIEW", "paraphrase; PDF has an Abstract heading but no clean cut"
        else:
            action, why = "REVIEW", "paraphrase?; no usable reference text"
    elif good_api:
        # a clean, verified abstract exists and the bib fails both T_HI against it and
        # T_ALIGN against the PDF: an edited or condensed copy, not extraction noise
        action, why, use = "REPLACE", f"edited (best {best:.2f}); {good_api[0]} abstract", good_api[0]
    else:
        action, why = "REVIEW", f"middle score {best:.2f}"
    replacement = (refs[use][0], use) if use else None

    rec["action"], rec["why"] = action, why
    if replacement:
        rec["replacement"], rec["replacement_src"] = replacement
    return rec


def abstract_sha(value):
    import hashlib

    return hashlib.sha1(re.sub(r"\s+", " ", value).strip().encode("utf-8")).hexdigest()


def load_reviewed():
    if os.path.exists(REVIEWED):
        with open(REVIEWED, encoding="utf-8") as f:
            return json.load(f)
    return {}


def cmd_score(bib, keys=None):
    text, entries = load(bib)
    reviewed = load_reviewed()
    recs = []
    for e in entries:
        if "abstract" not in e.fields or (keys and e.key not in keys):
            continue
        r = score_entry(e)
        hand = reviewed.get(e.key)
        if hand and r["action"] != "VERBATIM" and hand["sha1"] == abstract_sha(e.get("abstract")):
            r["auto_action"], r["auto_why"] = r["action"], r["why"]
            r["action"], r["why"] = "VERBATIM", "hand-checked: " + hand["note"]
            r.pop("replacement", None)
        recs.append(r)
    if keys and len(recs) < len(keys):
        found = {r["key"] for r in recs}
        print("no abstract in bib (nothing to audit): " + ", ".join(sorted(set(keys) - found)))
    os.makedirs(STATE, exist_ok=True)
    with open(REPORT, "w", encoding="utf-8") as f:
        json.dump(recs, f, ensure_ascii=False, indent=1)
    cols = ["key", "action", "best", "best_src", "align_pdf", "fwd", "rev", "api_in_pdf", "why", "flags", "candidate_src", "sources", "bib_head"]
    with open(REPORT[:-5] + ".tsv", "w", encoding="utf-8") as f:
        f.write("\t".join(cols) + "\n")
        for r in recs:
            row = []
            for c in cols:
                v = r.get(c, "")
                if isinstance(v, (dict, list)):
                    v = json.dumps(v, ensure_ascii=False, separators=(",", ":"))
                row.append(str(v).replace("\t", " ").replace("\n", " "))
            f.write("\t".join(row) + "\n")
    counts = {}
    for r in recs:
        counts[r["action"]] = counts.get(r["action"], 0) + 1
    print(json.dumps(counts))
    for r in recs:
        if keys or r["action"] != "VERBATIM":
            src = f" -> {r['replacement_src']}" if r.get("replacement_src") else ""
            print(f"{r['action']:8s} {r['key']}: {r['why']}{src}")
    print(f"report: {REPORT[:-5]}.tsv")


# --------------------------------------------------------------------------
# Applying changes
# --------------------------------------------------------------------------


def bib_escape(s):
    """Make plain text safe inside a braced BibTeX value without changing its words."""
    s = re.sub(r"\s+", " ", s).strip()
    # keep braces balanced: drop any unmatched ones
    out, depth = [], 0
    for c in s:
        if c == "{":
            depth += 1
        elif c == "}":
            if depth == 0:
                continue
            depth -= 1
        out.append(c)
    s = "".join(out)
    while depth > 0:
        i = s.rfind("{")
        s = s[:i] + s[i + 1 :]
        depth -= 1
    return s


def source_text(e, spec):
    """Text named by an override: {"from": "crossref"|"openalex"|"arxiv"|"cut:N"|"span",
    "start": ..., "end": ..., "edits": [[old, new], ...]}. "span" takes the PDF text
    from the first occurrence of `start` through the following `end`, inclusive,
    cleaned like a cut. Each edit must match exactly once."""
    src = spec["from"]
    doi = entry_doi(e)[0]
    if src == "crossref":
        t = fetch_crossref(doi)["abstract"]
    elif src == "openalex":
        t = fetch_openalex(doi)["abstract"]
    elif src == "arxiv":
        with open(cache_path("arxiv", arxiv_id(e)), encoding="utf-8") as f:
            t = json.load(f)["abstract"]
    else:
        with open(cache_path("pdf", e.key), encoding="utf-8") as f:
            d = json.load(f)
        if src.startswith("cut:"):
            cands = pdf_cuts(d.get("head") or "") + pdf_cuts(d.get("wide") or "")
            t = clean_pdf_cut(cands[int(src[4:])][0], d["full"])
        elif src == "span":
            full = d["full"]
            for k, v in LIGATURES.items():
                full = full.replace(k, v)
            loose = lambda x: re.compile(r"\s+".join(map(re.escape, x.split())))
            ms = loose(spec["start"]).search(full)
            me = ms and loose(spec["end"]).search(full, ms.start())
            if not (ms and me) or me.end() - ms.start() > 6000:
                raise SystemExit(f"span anchors not found (or span too long) in {e.key}")
            t = clean_pdf_cut(full[ms.start() : me.end()], full)
        else:
            raise SystemExit(f"unknown source {src} for {e.key}")
    if spec.get("drop_from"):
        # drop a trailing non-abstract block (funding, supplement, license lines)
        if t.count(spec["drop_from"]) != 1:
            raise SystemExit(f"drop_from marker matches {t.count(spec['drop_from'])} times in {e.key}")
        t = t[: t.index(spec["drop_from"])].rstrip()
    for old, new in spec.get("edits", []):
        if t.count(old) != 1:
            raise SystemExit(f"edit {old!r} matches {t.count(old)} times in {e.key}")
        t = t.replace(old, new)
    return t


def cmd_apply(bib, overrides=None, dry=False, keys=None):
    text, entries = load(bib)
    with open(REPORT, encoding="utf-8") as f:
        recs = {r["key"]: r for r in json.load(f)}
    manual = {}
    if overrides:
        with open(overrides, encoding="utf-8") as f:
            manual = json.load(f)  # key -> {"action": "KEEP"|"DELETE"|"REPLACE", "from": ..., "note": ...}
    edits = []  # (start, end, new)
    summary, unresolved = {}, []
    for e in entries:
        f = e.fields.get("abstract")
        if not f:
            continue
        if keys and e.key not in keys:
            continue
        r = recs.get(e.key)
        if r is None:
            if keys:
                raise SystemExit(f"{e.key} is not in the report; rerun score with the same --keys")
            continue  # the report covers only some entries (score --keys)
        if r["bib_head"] != re.sub(r"\s+", " ", f.value)[:120]:
            raise SystemExit(f"report is stale for {e.key}; rerun score")
        if e.key in manual:
            m = manual[e.key]
            act = m["action"]
            new = source_text(e, m) if act == "REPLACE" else None
            tag = act + "(manual)"
        elif r["action"] in ("REPLACE", "DELETE"):
            act, new, tag = r["action"], r.get("replacement"), r["action"]
        elif r["action"] == "REVIEW":
            unresolved.append(e.key)
            continue
        else:
            continue
        if act == "DELETE":
            edits.append((f.line_start, f.line_end, ""))
        elif act == "REPLACE":
            edits.append((f.vstart, f.vend, bib_escape(new)))
        elif act != "KEEP":
            raise SystemExit(f"bad action {act} for {e.key}")
        summary[tag] = summary.get(tag, 0) + 1
    if unresolved:
        raise SystemExit(f"{len(unresolved)} REVIEW entries without an override: {unresolved}")
    for s_, t_, new in sorted(edits, reverse=True):
        text = text[:s_] + new + text[t_:]
    print(json.dumps(summary, indent=1))
    if not dry:
        with open(bib, "w", encoding="utf-8") as fh:
            fh.write(text)


def cmd_preview(bib, overrides, key):
    """Print the text an override would write."""
    text, entries = load(bib)
    with open(overrides, encoding="utf-8") as f:
        m = json.load(f)[key]
    e = [x for x in entries if x.key == key][0]
    print(m["action"], m.get("note", ""))
    if m["action"] == "REPLACE":
        print(bib_escape(source_text(e, m)))


def cmd_show(bib, key, n=4000):
    """Print the bib abstract next to every source and PDF cut, for REVIEW."""
    text, entries = load(bib)
    e = [x for x in entries if x.key == key][0]
    recs = {r["key"]: r for r in json.load(open(REPORT, encoding="utf-8"))} if os.path.exists(REPORT) else {}
    r = recs.get(key, {})
    print(f"== {key} [{e.etype}] {r.get('action', '?')}: {r.get('why', 'not scored')}")
    print("title:", e.get("title"), "|", e.get("journal") or e.get("booktitle") or e.get("publisher"), e.get("year"))
    print("file:", e.get("file"), " doi:", e.get("doi"), (" doi by title: " + r["doi_found"]) if r.get("doi_found") else "")
    print("\nBIB:", re.sub(r"\s+", " ", e.get("abstract") or "(none)"))
    doi = entry_doi(e)[0]
    if doi:
        for k, fn in (("crossref", fetch_crossref), ("openalex", fetch_openalex)):
            t = fn(doi).get("abstract")
            if t:
                print(f"\n{k.upper()}:", t[:n])
    aid = arxiv_id(e)
    if aid and os.path.exists(cache_path("arxiv", aid)):
        print("\nARXIV:", json.load(open(cache_path("arxiv", aid), encoding="utf-8")).get("abstract", "")[:n])
    p = cache_path("pdf", key)
    if os.path.exists(p):
        d = json.load(open(p, encoding="utf-8"))
        cands = pdf_cuts(d.get("head") or "") + pdf_cuts(d.get("wide") or "")
        for i, (t, h) in enumerate(cands):
            print(f"\nCUT {i} [{h}]:", clean_pdf_cut(t, d["full"])[:n])
        if not cands:
            print("\nPDF: no cut found; read the first pages of", e.get("file"))


def cmd_reviewed(bib, key, note):
    """Record that KEY's current abstract was checked against the page by hand,
    so later full audits report it as VERBATIM until the text changes."""
    text, entries = load(bib)
    e = [x for x in entries if x.key == key][0]
    d = load_reviewed()
    d[key] = {"sha1": abstract_sha(e.get("abstract")), "note": note, "date": time.strftime("%Y-%m-%d")}
    os.makedirs(STATE, exist_ok=True)
    with open(REVIEWED, "w", encoding="utf-8") as f:
        json.dump(d, f, ensure_ascii=False, indent=1)
    print(f"recorded {key} as hand-checked")


def _opt(args, name):
    if name in args:
        i = args.index(name)
        v = args[i + 1]
        del args[i : i + 2]
        return v
    return None


if __name__ == "__main__":
    args = sys.argv[1:]
    if len(args) < 2:
        raise SystemExit(__doc__)
    keys = _opt(args, "--keys")
    keys = {k.strip() for k in keys.split(",") if k.strip()} if keys else None
    ov = _opt(args, "--overrides")
    dry = "--dry-run" in args
    args = [a for a in args if a != "--dry-run"]
    cmd, bib = args[0], os.path.expanduser(args[1])
    if cmd == "fetch":
        cmd_fetch(bib, keys)
    elif cmd == "score":
        cmd_score(bib, keys)
    elif cmd == "apply":
        cmd_apply(bib, ov, dry, keys)
    elif cmd == "preview":
        cmd_preview(bib, os.path.expanduser(args[2]), args[3])
    elif cmd == "show":
        cmd_show(bib, args[2])
    elif cmd == "reviewed":
        cmd_reviewed(bib, args[2], " ".join(args[3:]))
    else:
        raise SystemExit(__doc__)
interface:
  display_name: "Citation Add"
  short_description: "Add new PDFs in ~/org/pdfs to references.bib with verbatim abstracts"
  default_prompt: "Use $citation-add to add the new PDFs in ~/org/pdfs to my bibliography."
{{ .chezmoi.homeDir }}/.agents/skills/citation-add
citation-hunt
---
name: citation-hunt
description: Given a section of text, find citations for uncited claims by scanning the full bib, tracing upstream references of matched papers, then searching with world knowledge and the web. Pauses for confirmation before fetching anything.
user-invocable: true
allowed-tools: Bash, Read, Edit, WebSearch, WebFetch
---

# citation-hunt

Find citations for all claims in a section of text - both uncited and already-cited ones. Works in three chapters:

- **Chapter 1 - Bib scan:** Decompose the section into paragraph-level claims, then scan every entry in `references.bib` using titles, abstracts, and `annote` fields to find the best match for each claim.
- **Chapter 2 - Upstream references:** For every paper matched in Chapter 1, read its reference list and trace one level upstream to find additional supporting sources not already in the bib.
- **Chapter 3 - World knowledge + web:** For claims still uncovered or only weakly covered, use world knowledge and targeted web searches to find papers that should exist.

## Invocation

```
/citation-hunt [section] [--file path/to/file.org]
```

- `section`: a heading name (partial match OK) or line range like `100-120`
- `--file`: path to the org/tex file to search. If omitted, defaults to `~/org/thesis/thesis.org`

The `--seeds` argument has been removed. Chapter 1 scans the entire bib. Chapter 2 traces upstream from whatever Chapter 1 finds.

---

## Phase 0: Write scratch file

Before doing anything else, establish a scratch directory:

```bash
mkdir -p ~/.cache/citation-hunt/scratch
```

After extracting the section (Phase 1b), immediately write the scratch file. The file persists across all phases and is the single source of truth for section text and claim state - read it rather than relying on in-context repetition when context grows long.

File path: `~/.cache/citation-hunt/scratch/<section-slug>.org`

The slug is the heading text lowercased, spaces replaced with hyphens, non-alphanumeric stripped (e.g., "Problem Statement" -> `problem-statement`).

File structure:

```org
* SECTION: <heading> (lines <start>-<end>, <filename>)
** Para 1 (lines <A>-<B>)
<verbatim paragraph text>

** Para 2 (lines <C>-<D>)
<verbatim paragraph text>

...

* CLAIMS
(written after Phase 2 - leave blank until then)

* TO-DO
(written after Phase 3 examiner judgment - one checkbox per paper per claim)
Format:
** TODO Para N - ClaimX description
   - [ ] @citekey (conf NN) - one line reason
   - [ ] @citekey2 (conf NN) - one line reason

Mark [ ] -> [X] as each paper is fully processed (roam + thesis.org both done).
**This list is the authoritative work queue for Phase 3.5.** Read it at the start
of each paper's processing cycle rather than reconstructing from context.

* MATCH STATE
(written incrementally during Chapter 1 - leave blank until then)
```

Delete this file at the end of Phase 6.

---

## Phase 1: Extract and confirm target text

### 1a. Locate the file

If `--file` was given, use that path. Otherwise use `~/org/thesis/thesis.org`.

### 1b. Extract the section

Read the file. Find the section matching the given heading (case-insensitive partial match) or line range.

Extract:
- The heading text
- All body text under that heading, stopping at the next same-level or higher heading
- Any existing cite keys already present (org `[[cite:@key]]` or LaTeX `\cite{key}` format)

**Segment by paragraph:** Split the extracted body text at blank lines. Each non-empty block is a paragraph. Record the line range for each paragraph.

Write the scratch file now (Phase 0 format above, CLAIMS and MATCH STATE sections left blank).

### 1c. Confirm with user

Print the extracted text verbatim in a code block, with paragraph boundaries marked. Then print:

```
--- citation-hunt: Phase 1 confirmation ---
File:    <resolved file path>
Section: <heading>
Lines:   <start>-<end>
Paragraphs found: <N>
Existing citations found: <list or "none">
Scratch file: ~/.cache/citation-hunt/scratch/<slug>.org

Proceed? (If this is wrong, stop and re-invoke with correct arguments.)
```

**STOP HERE.** Do not proceed to Phase 2 until the user confirms. If the text is wrong, abort and delete the scratch file.

---

## Phase 2: Decompose into claims

Read the scratch file. For each paragraph, break it into individual claims. A claim is any sentence or clause that:
- States a fact about the world ("software systems degrade over time")
- Makes a motivation or justification ("this is problematic because...")
- States a prevalence or frequency ("decisions are rarely documented")
- Attributes a property to a concept ("architecture erosion causes...")

Skip: purely definitional sentences that introduce your own framework/terminology (those don't need external citations), and transition sentences.

**Do NOT skip already-cited sentences.** Process every claim regardless of whether it already has a citation - the goal is the *best* citation, not just any citation.

For each claim, record:
- **Para**: which paragraph it came from (number and line range)
- **Type**: one of:
  - `count/prevalence` - asserts how often, how many, or how common something is. Needs a measured quantity, survey count, or documented frequency. Example: "architects make many decisions", "documentation is rarely maintained"
  - `causal/consequence` - asserts that X causes Y or that a phenomenon produces a specific outcome. Needs a mechanism, longitudinal study, or controlled evidence. Example: "knowledge loss increases maintenance costs"
  - `canonical/definitional` - introduces an established concept or framework. Needs the paper that coined or canonically established it. Example: "bounded rationality", "architecture erosion", "residual causality"
  - `cognitive/behavioral` - claims about how people think, decide, or behave. Needs psychology, HCI, or cognitive science study. Example: "experts rely on pattern recognition", "decision fatigue impairs judgment"
  - `observation-in-practice` - asserts that something happens in real software projects. Needs a case study, interview study, or empirical observation in actual projects. Example: "ADRs are rarely written", "architects face time pressure"
  - `design-rationale` - justifies a specific approach or tool. Needs prior work proposing or evaluating that approach. Example: "ADRs reduce knowledge vaporization"

Also record for each claim: the **minimum evidence bar** - one sentence describing what a PhD examiner would require to accept a citation for this claim. Be specific. Example: "a study that counted or surveyed the number of architectural decisions made per project" - not just "a paper about architectural decisions."

Write the CLAIMS section of the scratch file:

```org
* CLAIMS
** Para 1 (lines <A>-<B>)
- [count/prevalence | bar: study measuring decision frequency or volume per project] architects make many decisions
- [causal/consequence | bar: empirical evidence linking undocumented decisions to maintenance cost increase] knowledge loss increases costs

** Para 2 (lines <C>-<D>)
- [canonical/definitional | bar: paper that coined or canonically defined the concept] architecture erosion is divergence between intended and actual structure
...
```

Print the decomposed claim list for visibility. Continue automatically (no pause needed here).

---

# CHAPTER 1: Full bib scan

## Phase 3: Match all claims against the full bibliography

### 3a. Load bib

Read `~/org/bib/references.bib`. For each entry, collect:
- citekey
- title
- abstract (if present)
- `annote` field (concise summary - use this as the primary matching signal when present)
- keywords (if present)
- `file` field (whether a local PDF exists)

### 3b. Ideal source reflection

Before scanning the bib, for each claim write a one-paragraph ideal source description. Answer:
- What methodology would this paper use? (e.g. controlled study, survey of N practitioners, systematic literature review, longitudinal case study, RCT, theoretical framework with empirical validation)
- What would its key finding or contribution look like? (e.g. "found that architects make on average X decisions per project", "defined and operationalized the concept of Y", "showed through interviews that Z is common in practice")
- What would disqualify a paper from satisfying this claim? (e.g. "a paper that only argues for X without measuring it", "a position paper", "a paper about a related but distinct phenomenon")

This description is your matching target for the bib scan - not the claim text itself.

Example for claim "architects make many decisions" (type: count/prevalence):
> Ideal: an empirical study - observational, survey, or case study - that measured or documented the volume of architectural decisions made during a software project. Key finding would be a count, frequency, or documented pattern of decision-making activity. Disqualifies: any paper that asserts decision volume without measuring it, position papers arguing for decision-centric architecture, conceptual frameworks.

### 3c. Match every claim against every bib entry

Process claims **one paragraph at a time**. After finishing all claims in a paragraph, immediately append those rows to the `* MATCH STATE` section of the scratch file before moving to the next paragraph. This ensures compaction cannot wipe in-progress work.

**Resuming after compaction:** If context compaction occurs mid-scan, re-read the scratch file at the start of the next message. Find the last paragraph recorded in `* MATCH STATE`. Resume scoring from the *next* paragraph - do not re-score paragraphs already written to MATCH STATE. Any paragraph whose rows are not yet in MATCH STATE must be re-scored from scratch (its in-progress work was lost).

For each claim, scan all bib entries and score them against the **ideal source description** from 3b - not the claim text. Use the `annote` as the primary signal. Fall back to abstract and title if `annote` is absent.

Scoring guidance:
- **direct (80-100):** The `annote` or abstract explicitly states the same phenomenon the claim asserts. The paper is clearly about this.
- **indirect (40-79):** The `annote`/abstract covers a related concept that supports the claim but is not the central point.
- **weak (10-39):** Tangential overlap - the paper is in the area but the claim is peripheral to what it studies.
- **none (0-9):** No meaningful overlap.

Hard disqualification rules by claim type - apply these before scoring. A paper that fails a disqualification rule is capped at score 39 regardless of topical overlap:
- `count/prevalence`: disqualify papers whose annote/abstract contains none of: "found that", "measured", "survey of", "interviewed", "N projects", "N architects", "empirical", "case study", "% of", "on average". Position papers, opinion pieces, and conceptual frameworks are disqualified.
- `causal/consequence`: disqualify papers that only assert the causal relationship without providing a mechanism, longitudinal evidence, or controlled study.
- `canonical/definitional`: survey and foundational papers are explicitly preferred. Do NOT disqualify.
- `cognitive/behavioral`: disqualify papers from SE venues that only discuss the phenomenon without grounding in psychology, cognitive science, or HCI study methodology.
- `observation-in-practice`: disqualify papers that are purely theoretical or that study student/toy projects rather than real-world software development.
- `design-rationale`: prefer papers that evaluate or propose the approach empirically; deprioritize papers that merely mention it.

For each claim, select the best candidate(s). A claim may have multiple good matches - list up to 3, ranked by score. Only list a candidate if score >= 40. Weak matches (10-39) are noted but not recommended.

### 3d. Examiner judgment

For each claim-candidate pair with score >= 40, apply an explicit PhD examiner judgment. Ask: *"If a PhD examiner saw this citation for this claim, would they accept it as genuine evidence, or would they push back?"*

Classify each pair as one of:

- **`DIRECT_EVIDENCE`**: the paper directly provides the kind of evidence the claim requires (matches the minimum evidence bar from Phase 2). A passage in this paper will satisfy a PhD examiner. Proceed to Phase 3.5.
- **`UPSTREAM_NEEDED`**: the paper is topically relevant and likely cites the actual primary evidence, but is itself a position paper, framing paper, survey, or opinion piece that doesn't directly evidence the claim. The real value is in its reference list. **Skip Phase 3.5 for this paper. Add to Chapter 2 seed list immediately.**
- **`REJECTED`**: on reflection, this paper doesn't actually support this claim even indirectly. Score downgraded to < 40. Move to uncovered.

The key distinction between `DIRECT_EVIDENCE` and `UPSTREAM_NEEDED`: ask "if I find a passage in this paper, will it contain a measurement, a documented observation, or an established definition - or will it just restate the same claim without grounding?" If the latter, it is `UPSTREAM_NEEDED`, not `DIRECT_EVIDENCE`.

**After completing each paragraph's claims:** append the new rows to the `* MATCH STATE` section of the scratch file before moving to the next paragraph. Do not wait until all paragraphs are done.

**Then** print the match table:

```
| Para | Claim (truncated 60 chars) | Type | Existing | Candidate | Conf | Judgment | Basis |
```

- **Judgment**: `DIRECT_EVIDENCE`, `UPSTREAM_NEEDED`, or `REJECTED`
- **Basis**: one sentence explaining the judgment

Claims flagged `UPSTREAM_NEEDED` or `REJECTED` with no `DIRECT_EVIDENCE` alternative: `[→ Chapter 2 seed: @key]`
Claims with no candidate at all (conf < 40): `[UNCOVERED - Chapter 2/3]`

Continue automatically.

---

## Phase 3.5: Locate supporting passages and page numbers

**Before starting:** write the `* TO-DO` section of the scratch file - one checkbox per `DIRECT_EVIDENCE` paper per claim. This is the authoritative work queue. At the start of each paper's processing cycle, re-read the scratch file's TO-DO list to find the next unchecked item. Mark `[ ]` -> `[X]` only after both roam annotation and thesis.org insertion are complete.

For each claim-citekey pair that is `DIRECT_EVIDENCE` AND `conf >= 60` AND has a local PDF, find the specific supporting passage. Skip any pair marked `UPSTREAM_NEEDED` or `REJECTED` - those go to Chapter 2.

**Process one paper at a time.** Complete the full cycle for a single paper (passage search -> feedback loop evaluation -> roam update -> scratch file update -> thesis.org insertion) before moving to the next. Do not batch multiple PDFs in parallel. This preserves the feedback loop and prevents context from being flooded with raw PDF text.

**CRITICAL - compaction safety:** After reading PDF text and identifying a passage, immediately write the roam file annotation AND insert the cite+transclusion block into thesis.org in the same response, before doing anything else. Never defer these writes to a later message. Context compaction can occur at any time between messages, wiping PDF text from context. If the writes are deferred, the work is lost. The sequence within a single message must be: (1) bbox, (2) create/update roam file, (3) insert cite+transclude into thesis.org. All three steps must complete before moving on to the next paper.

### 3.5a. Read roam file and check for local PDF

**First:** check if `~/org/roam/refs/<citekey>.org` exists and read it. Scan all existing `citation-hunt annotations` subheadings for passages that already match the current claim. If a match exists:
- Reuse the existing annotation (cite key + heading) immediately - emit the `[cite:]` and `#+transclude:` block into thesis.org right away
- Then continue searching the PDF for *additional* passages that the existing annotation does not cover

**Then:** read the `file` field from the bib entry. If no `file` field, skip - report `p. ?` and continue.

### 3.5b. Choose navigation strategy by page count

```bash
pdfinfo <file> | grep Pages
```

- **<= 20 pages:** Path A (full-read) - extract all text at once
- **21-60 pages:** Path A (search-first) - search by term across all pages, then read ±2 pages around each hit for context
- **> 60 pages (books/theses):** Path B (toc-first) - read first 15 pages for TOC, identify relevant sections, read only those

### 3.5c. Extract search terms from the abstract/annote

Do NOT use words from the claim - the claim is a paraphrase, not a quote from the paper. Instead:

1. Read the bib entry's `annote` field (preferred) or `abstract` field
2. Identify 2-3 distinctive noun phrases that are specific to the point supporting the claim
   - Good: "knowledge vaporization", "design decisions not captured", "documentation entropy"
   - Bad: "software", "system", "architecture" (too common)
3. These are your search terms - words the paper itself uses when making its point

### 3.5d. Find all supporting passages

For each claim, find **all** passages in the PDF that support it - not just the first match.

**Path A - short docs (full-read):**

```bash
python3 -c "
import subprocess
text = subprocess.run(['pdftotext', 'FILEPATH', '-'], capture_output=True, text=True).stdout
pages = text.split('\f')
query = 'SEARCH TERM'
for i, page in enumerate(pages, 1):
    hits = [l.strip() for l in page.splitlines() if query.lower() in l.lower() and len(l.strip()) > 50]
    if hits:
        print(f'p.{i}:', hits[0][:120])
"
```

Collect all hits across all terms. For each hit page, read a few lines of context to extract the full supporting sentence(s). Discard hits that are: reference list entries, section headings only, figure captions, or repetitions of the same argument already captured from another page.

**Path B - long docs (toc-first):**

1. Read the first 15 pages for the TOC
2. Identify **all** candidate sections whose titles relate to the claim's topic
3. Read each candidate section range and collect all substantive passages
4. Do not stop at the first hit - work through all relevant sections

Aim for 2-4 distinct passages per claim-citekey pair. One passage is acceptable for short papers or when the argument is made in only one place.

**Feedback loop:** After reading the passage, apply the PhD examiner test: does this passage contain a specific piece of evidence - a measurement, a documented observation, a named study result, or a canonical definition? A passage that merely restates the claim or asserts the same thing without grounding is NOT evidence and must be rejected.

Concretely: if the passage could have been written by the thesis author themselves without any empirical work, it is not evidence. Reject it.

If the retrieved passage fails:
- Search for other passages in the same paper that do contain grounding
- If found, use that instead
- If no passage in this paper provides grounding, downgrade to `UPSTREAM_NEEDED` and add to Chapter 2 seed list - do not emit a citation for this paper
- Update MATCH STATE in the scratch file

### 3.5e. Update roam ref file

For each citekey, create the roam ref file if needed, then add one subheading per passage found:

1. Check if `~/org/roam/refs/<citekey>.org` exists
2. If not, create it:
   ```bash
   python3 ~/.agents/skills/citation-hunt/create_ref.py \
     --citekey <key> \
     --title "<title from bib>" \
     --pdf "<file from bib>"
   ```
3. For each passage, run bbox.py to write the green highlight and get coordinates:
   ```bash
   python3 ~/.agents/skills/citation-hunt/bbox.py \
     --pdf "<file from bib>" \
     --page <N> \
     --color green \
     --quote "<verbatim passage>"
   ```
   bbox.py runs a four-tier extraction ladder automatically:
   - Tier 1: `pdftotext -bbox-layout` + XML parse (precise word bboxes)
   - Tier 2: XML sanitization + retry (handles control chars / bare ampersands)
   - Tier 3: pypdf visitor API (char-level positions, zero extra deps)
   - Tier 4: pdfminer.six (best encoding coverage)
   - Tier 5: `pdftoppm` visual render - exits with code 2 and prints `VISUAL_FALLBACK: /tmp/.../page.png`

   Capture stdout: `NOTER_PAGE: (N x . y)` and `HIGHLIGHT: #s(pdf-highlight ...)`.

   If exit code is 2 (VISUAL_FALLBACK): read the PNG path from stderr, use the Read
   tool to view the image, visually locate the passage, estimate fractional x/y coords
   (0.0=left/top, 1.0=right/bottom), then use `(N x . y)` as the NOTER_PAGE manually.
   No highlight is written to the PDF in this case - note it as `p.N (visual approx)`.

   If exit code is 1 (MATCH_FAILED): fall back to `(N 0.5 . 0.5)` with no HIGHLIGHT.

4. Append a `* citation-hunt annotations` section if not already present. Add one subheading per passage.
   Keep the heading text exactly `citation-hunt annotations`: the site build (`lisp/preamble.el`,
   `lisp/project.el`) looks for it.

```org
* citation-hunt annotations
:PROPERTIES:
:GENERATED: yes
:LAST_UPDATED: <date>
:END:

** <inferred heading summarizing the passage>
:PROPERTIES:
:NOTER_PAGE: (N x . y)
:HIGHLIGHT: #s(pdf-highlight N ((y1 x1 y2 x2)))
:CITE: [cite:@<key> p. <N>]
:END:
#+BEGIN_QUOTE
<verbatim passage>
#+END_QUOTE
```

Do not duplicate passages already present in the roam file from a previous run - check existing subheadings before appending.

### 3.5f. Format the citations in thesis.org

One cite+transclusion block per passage. Each passage in the roam file gets its own transclusion line.

```org
[cite:@key p. N]
#+transclude: [[file:~/org/roam/refs/<citekey>.org::*<heading for passage at p.N>]]
[cite:@key p. M]
#+transclude: [[file:~/org/roam/refs/<citekey>.org::*<heading for passage at p.M>]]
```

Rules:
- If a page is found: use `[cite:@key p. N]` form
- If no match found (scanned PDF, encrypted): use `[cite:@key]`, note `p. ? (text search failed)` in report
- For entries with no `file` field: use `[cite:@key]` and skip roam annotation

### 3.5g. Report

Update the match table from Phase 3c with a "Pages" column (listing all passages found):

```
| Para | Claim (truncated) | Type | Existing | Candidate | Conf | Pages | Basis |
```

Continue automatically.

---

# CHAPTER 2: Upstream reference tracing

Run this chapter for:
- Claims still uncovered (conf < 40) after Chapter 1
- Claims with only weak/indirect matches (conf 40-59) that warrant a stronger source
- Claims where the best Chapter 1 match was judged `UPSTREAM_NEEDED` - regardless of its confidence score. A high-confidence topical match that can't directly evidence the claim is a seed, not a citation.

## Phase 4: Trace upstream references from Chapter 1 matches

### 4a. Identify papers to trace

Collect all citekeys that were matched in Chapter 1 with conf >= 40. These are the seed papers for upstream tracing. Also include any citekeys already cited in the section.

Do not limit to a curated seed list - trace upstream from everything Chapter 1 found.

### 4b. Read seed paper reference lists

For each seed paper that has a local PDF (check `file` field in the bib entry):
1. Read the last 2-4 pages of the PDF to get the reference list
2. For each reference, note: authors, title, year, venue
3. Flag references that are clearly relevant to the *uncovered or weakly-covered claims* based on title alone

Depth is **1 level**: read the reference lists of the matched papers only, not the reference lists of those references.

Skip papers already in `~/org/bib/references.bib` (they were already scanned in Chapter 1).

### 4c. Search for open-access PDFs

Read `~/.agents/skills/citation-hunt/sources.md`. Try each source in the listed order.

For each flagged upstream reference not already in `~/org/pdfs/`:
1. Work through the sources in `sources.md` in order until a PDF is found or all are exhausted
2. Do NOT attempt paywalled downloads (IEEE Xplore, ACM DL, Springer without open access)
3. If found via a source not in `sources.md`, append it to the "Additional sources found during hunting" section of `sources.md`
4. Note the URL but do NOT download yet

### 4d. Confirm upstream candidates with user

Print a structured report:

```
--- citation-hunt: Chapter 2 confirmation ---

Claims targeted (uncovered or weak from Chapter 1):
  1. [Para N] "<claim text>" (current best: @key conf=NN, or "none")
  2. ...

Upstream candidates found:

  For claim 1:
    - [TITLE] Author et al. YEAR (VENUE)
      Why: <one sentence>
      PDF: <URL or "paywalled - manual download needed">

  For claim 2:
    - ...

Actions if approved:
  - Download N PDFs to ~/org/pdfs/
  - Run /citation-add to generate bib entries
  - Re-run passage search on new papers

Approve downloads? List any to skip.
```

**STOP HERE.** Do not download anything until the user approves.

### 4e. Fetch approved PDFs and process

For each approved candidate with a found PDF URL:
1. Download to `~/org/pdfs/` using curl:
   ```bash
   curl -L "<url>" -o "~/org/pdfs/<authorYEARkeyword>.pdf"
   ```
2. Verify the file is a valid PDF:
   ```bash
   pdfinfo ~/org/pdfs/<filename>.pdf 2>/dev/null | head -3
   ```
3. If verification fails, delete the file and report as failed.

For paywalled papers, print the title and DOI so the user can download manually.

After fetching, run Phase 3.5 passage search on each new PDF immediately to check whether it actually supports the claim before proceeding to Chapter 3.

Update the MATCH STATE section of the scratch file with any new matches found.

---

# CHAPTER 3: World knowledge and web search

Run this chapter for claims still uncovered or weakly covered after Chapters 1 and 2.

## Phase 5: World knowledge pass

### 5a. Identify remaining gaps

Re-read the MATCH STATE section of the scratch file. List all claims with conf < 60 after Chapters 1 and 2.

### 5b. Apply world knowledge

For each remaining uncovered or weak claim, reason from your training knowledge about what papers are likely to exist that support it. Consider:
- Classic papers in the field that are commonly cited for this type of claim
- Foundational works the claim implicitly relies on
- Empirical studies you know of that measured this phenomenon

For each suggestion, produce:
- Full title (as accurate as possible)
- Author(s) and year (if known with reasonable confidence)
- Venue (if known)
- Why it supports the claim
- Confidence in the suggestion: `high` (paper is well-known and you are sure it exists), `medium` (you believe it exists but may have detail errors), `low` (this type of paper should exist; treat as a search lead)

Flag `medium` and `low` confidence suggestions explicitly - the user should verify before citing.

### 5c. Web search pass

For each remaining gap, perform targeted web searches to find papers not in your training or with uncertain details.

Use queries such as:
- `"<key phrase from claim>" software engineering empirical study site:semanticscholar.org`
- `"<phenomenon>" architecture decisions survey filetype:pdf`
- Semantic Scholar API: `https://api.semanticscholar.org/graph/v1/paper/search?query=<terms>&fields=title,authors,year,openAccessPdf,abstract`

For each web result:
- Verify the paper actually exists (check title + authors on Semantic Scholar or the publisher page)
- Do NOT fabricate or guess DOIs
- Note whether an open-access PDF is available

### 5d. Confirm Chapter 3 candidates with user

```
--- citation-hunt: Chapter 3 confirmation ---

Remaining uncovered claims after Chapters 1-2:
  1. [Para N] "<claim text>"

World knowledge suggestions:
  For claim 1:
    - [TITLE] Author et al. YEAR - confidence: high/medium/low
      Why: <one sentence>
      Note: <any caveats about title/year accuracy>

Web search finds:
  For claim 1:
    - [TITLE] Author et al. YEAR (VENUE)
      PDF: <URL or "paywalled">
      Verified: yes/no

Actions if approved:
  - Download N PDFs
  - Run /citation-add

Approve? List any to skip.
```

**STOP HERE.** Do not download until approved.

### 5e. Fetch approved Chapter 3 PDFs

Same procedure as Phase 4e. Run Phase 3.5 passage search on each new PDF.

Update the MATCH STATE section of the scratch file.

---

## Phase 6: Final report and cite placement

### 6a. Paragraph-grouped placement report

Group all recommendations by paragraph. This is the primary output:

```
--- citation-hunt: Results ---

Para 1 (lines <A>-<B>):
  "<sentence>" -> @key p.N (conf: 90, direct)
    Passage: "<verbatim supporting sentence from paper>"

  "<sentence>" -> @key2 (conf: 72, indirect) | no local PDF
    Basis: <annote/abstract sentence>

Para 2 (lines <C>-<D>):
  "<sentence>" -> @key3 p.4 (conf: 85, direct)
    Passage: "..."

  "<sentence>" -> UNCOVERED
    Best found: @key4 (conf: 35, weak) - not recommended
    Suggestion: look for empirical study on X

...

BETTER ALTERNATIVES FOUND:
  Line <N>: currently cited @old - recommend @new instead (conf 55 -> 88)
  Basis: <why new is stronger>

PDFS DOWNLOADED (pending /citation-add):
  <filename>

PDFS NEEDING MANUAL DOWNLOAD:
  <title> | DOI: <doi>

NEXT STEPS:
  1. Run /citation-add to generate bib entries for new PDFs
  2. Review paragraph-level placements above
  3. For uncovered claims, consider manual library search
```

### 6b. Emit thesis.org cite blocks

For each confirmed match, emit the cite+transclusion block as specified in Phase 3.5f, grouped by paragraph and in reading order.

### 6c. Cleanup

Delete the scratch file:

```bash
rm ~/.cache/citation-hunt/scratch/<slug>.org
```

---

## Important rules

- Never cite a survey paper for a claim that the survey itself attributes to a primary source - flag it and suggest the primary source instead.
- Never invent DOIs, URLs, or quotes. If you can't find a direct quote from the abstract supporting the claim, say so.
- Never download from paywalled sources.
- The verbatim abstract requirement is enforced by `/citation-add`, not this skill - just get the PDFs into `~/org/pdfs/`.
- Process every claim regardless of whether it already has a citation. The goal is the best possible citation, not just any citation.
- The scratch file is the source of truth across phases. When in doubt, re-read it rather than reconstructing from context.
- A candidate from world knowledge with `medium` or `low` confidence must be flagged as unverified in the final report. Never emit a `[cite:]` for an unverified paper.
- Chapter 3 web searches are a last resort for genuinely uncovered claims. Do not run them for claims already covered at conf >= 60.
# citation-hunt: PDF Search Sources

When searching for open-access PDFs in Phase 4c, try these sources in order.
If a source not listed here yields a successful hit, append it to this file.

## Primary sources (try first)

1. **arXiv** - `https://arxiv.org/abs/<id>` or search `https://arxiv.org/search/?query=<title>&searchtype=all`
   - Best for: CS/SE preprints, recent papers
   - PDF URL pattern: `https://arxiv.org/pdf/<id>`

2. **Semantic Scholar** - `https://api.semanticscholar.org/graph/v1/paper/search?query=<title>&fields=title,authors,year,openAccessPdf`
   - Best for: finding open-access PDF links by title search; returns `openAccessPdf.url` if available
   - No auth required for basic search

3. **Unpaywall** - `https://api.unpaywall.org/v2/<doi>?email=citation-hunt@thesis`
   - Best for: finding legal open-access versions of paywalled papers when you have a DOI
   - Returns `best_oa_location.url_for_pdf` if an OA version exists

4. **OpenAlex** - `https://api.openalex.org/works?search=<title>&select=title,open_access,primary_location`
   - Best for: broad coverage, checking if a paper has any OA version
   - Returns `open_access.oa_url` if available

5. **SEI CMU library** - `https://resources.sei.cmu.edu/library/`
   - Best for: SEI technical reports, ATAM, architecture evaluation papers

6. **Author institutional pages** - search `<author name> <paper title> site:<university>.edu filetype:pdf`
   - Best for: papers where author self-hosts a preprint

## Fallback (try if all above fail)

- WebSearch with query: `"<exact title>" filetype:pdf`
- WebSearch with query: `"<first author lastname>" "<year>" "<first significant title word>" PDF`

## How to add a new source

If a source not listed here yields a PDF that downloads and validates successfully,
append a new entry under "## Additional sources found during hunting" below,
following the same format: name, URL pattern, best-for note.

## Additional sources found during hunting

<!-- append new entries here -->
#!/usr/bin/env python3
"""
Create a new org-roam ref file for a citekey, matching the citar-org-roam
capture template defined in doom config.

Usage:
    python3 create_ref.py --citekey KEY --title TITLE --pdf ~/org/pdfs/file.pdf

Output:
    CREATED:<path>   - file was created
    EXISTS:<path>    - file already exists, not overwritten
    ERROR:<reason>

The file is registered in the org-roam DB via emacsclient after creation.
"""

import sys
import os
import subprocess
import argparse
from datetime import datetime

ROAM_REFS_DIR = os.path.expanduser("~/org/roam/refs")

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--citekey", required=True)
    parser.add_argument("--title", required=True)
    parser.add_argument("--pdf", required=True)
    args = parser.parse_args()

    out_path = os.path.join(ROAM_REFS_DIR, f"{args.citekey}.org")

    if os.path.exists(out_path):
        print(f"EXISTS:{out_path}")
        return

    pdf_path = os.path.expanduser(args.pdf)
    date_str = datetime.now().strftime("%Y-%m-%d %a")

    content = f""":PROPERTIES:
:NOTER_DOCUMENT: {pdf_path}
:ROAM_REFS: @{args.citekey}
:END:
#+title: {args.title}
#+DATE: <{date_str}>
#+PUBLISH: reading

* Summary
-

* Key Points
-

* Annotations
"""

    with open(out_path, "w") as f:
        f.write(content)

    # Register in org-roam DB
    try:
        subprocess.run(
            ["emacsclient", "--eval",
             f'(org-roam-db-update-file "{out_path}")'],
            capture_output=True, timeout=10
        )
    except Exception:
        pass  # DB update failure is non-fatal - roam will catch it on next sync

    print(f"CREATED:{out_path}")


if __name__ == "__main__":
    main()
#!/usr/bin/env python3
"""
Find the bounding box of a quote on a PDF page and write a highlight annotation.

Usage:
    python3 bbox.py --pdf <path> --page <N> --quote "<text>" [--color green]

Output (on success, to stdout):
    NOTER_PAGE: (N x1 . y1)
    HIGHLIGHT: #s(pdf-highlight N ((y1 x1 y2 x2)))

Also writes a highlight annotation directly into the PDF at the matched word spans.
Color is an RGB tuple name: green (citation-hunt), yellow (default manual).

On MATCH_FAILED: exits with code 1 and prints reason to stderr.

Extraction fallback ladder:
  1. pdftotext -bbox-layout  -> precise word bboxes via XHTML
  2. XML sanitize + retry     -> handles control chars / unescaped & in text layer
  3. pypdf visitor API        -> char-level positions, zero extra deps
  4. pdfminer.six             -> best encoding coverage, handles more PDF variants
  5. pdftoppm visual render   -> page rendered to PNG, coords estimated from image read
  6. MATCH_FAILED             -> all extraction methods exhausted
"""

import argparse
import os
import re
import subprocess
import sys
import tempfile
import xml.etree.ElementTree as ET

COLORS = {
    'green':  (0, 1, 0),
    'yellow': (1, 1, 0),
    'cyan':   (0, 1, 1),
}


# ---------------------------------------------------------------------------
# Normalization (shared by all tiers)
# ---------------------------------------------------------------------------

def normalize(s):
    s = re.sub(r'-', ' ', s.lower())
    return re.sub(r'[^\w\s]', '', s).split()


# ---------------------------------------------------------------------------
# Span matching (shared by all tiers)
# words: list of (text, x1, y1_top, x2, y2_top) in top-left origin coords
# Returns matched word sublist or raises ValueError.
# ---------------------------------------------------------------------------

def match_span(words, quote):
    q_words = normalize(quote)
    if len(q_words) < 4:
        raise ValueError("Quote too short for reliable matching (< 4 words)")

    anchor_start = q_words[:4]
    anchor_end   = q_words[-4:]

    flat = []  # (norm_word, word_idx)
    for wi, w in enumerate(words):
        for nw in normalize(w[0]):
            flat.append((nw, wi))

    if len(flat) < 8:
        raise ValueError("Too few words on page for matching")

    start_flat = None
    for i in range(len(flat) - 4):
        span = [flat[i + j][0] for j in range(4)]
        if sum(a == b for a, b in zip(span, anchor_start)) >= 3:
            start_flat = i
            break
    if start_flat is None:
        raise ValueError("Start anchor not found")

    end_flat = None
    search_limit = min(start_flat + len(q_words) + 80, len(flat) - 4)
    for i in range(start_flat, search_limit):
        span = [flat[i + j][0] for j in range(4)]
        if sum(a == b for a, b in zip(span, anchor_end)) >= 3:
            end_flat = i + 3
            break
    if end_flat is None:
        raise ValueError("End anchor not found")

    start_wi = flat[start_flat][1]
    end_wi   = flat[end_flat][1]
    return words[start_wi:end_wi + 1]


# ---------------------------------------------------------------------------
# Tier 1: pdftotext -bbox-layout
# ---------------------------------------------------------------------------

def _sanitize_xml(raw):
    """Strip control characters and fix bare ampersands that break XML parsing."""
    cleaned = re.sub(r'[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]', '', raw)
    cleaned = re.sub(r'&(?!(?:[a-zA-Z][a-zA-Z0-9]*|#[0-9]+|#x[0-9a-fA-F]+);)', '&amp;', cleaned)
    return cleaned


def tier1_pdftotext(pdf_path, page):
    """Returns (words, W, H) with top-left origin coords, or raises RuntimeError."""
    result = subprocess.run(
        ['pdftotext', '-f', str(page), '-l', str(page), '-bbox-layout', pdf_path, '-'],
        capture_output=True, text=True
    )
    if result.returncode != 0 and not result.stdout.strip():
        raise RuntimeError(f"pdftotext failed: {result.stderr.strip()}")

    raw = result.stdout

    # Try parse as-is, then with sanitization
    root = None
    for attempt, xml_str in enumerate([raw, _sanitize_xml(raw)], 1):
        try:
            root = ET.fromstring(xml_str)
            break
        except ET.ParseError as e:
            if attempt == 2:
                raise RuntimeError(f"XML parse failed after sanitization: {e}")

    NS = 'http://www.w3.org/1999/xhtml'
    page_el = root.find(f'.//{{{NS}}}page')
    if page_el is None:
        raise RuntimeError("No <page> element in pdftotext bbox output")

    W = float(page_el.get('width'))
    H = float(page_el.get('height'))
    words = [
        (w.text or '', float(w.get('xMin')), float(w.get('yMin')),
         float(w.get('xMax')), float(w.get('yMax')))
        for w in root.iter(f'{{{NS}}}word')
    ]
    if not words:
        raise RuntimeError("pdftotext returned zero words")
    return words, W, H


# ---------------------------------------------------------------------------
# Tier 3: pypdf visitor API
# coords from pypdf are PDF-native (bottom-left origin); we flip to top-left.
# ---------------------------------------------------------------------------

def tier3_pypdf(pdf_path, page):
    """Returns (words, W, H) with top-left origin coords, or raises RuntimeError."""
    from pypdf import PdfReader

    reader = PdfReader(pdf_path)
    if page < 1 or page > len(reader.pages):
        raise RuntimeError(f"Page {page} out of range (doc has {len(reader.pages)} pages)")

    page_obj = reader.pages[page - 1]
    mb = page_obj.mediabox
    W = float(mb.width)
    H = float(mb.height)

    # Visitor yields (text, cm, tm, fontname, fontsize)
    # tm[4]=x, tm[5]=y in PDF coords (bottom-left origin)
    chunks = []  # (text, x_bl, y_bl, fontsize)

    def visitor(text, cm, tm, font, size):
        if text and text.strip():
            x, y = float(tm[4]), float(tm[5])
            sz = float(size) if size else 10.0
            chunks.append((text, x, y, sz))

    page_obj.extract_text(visitor_text=visitor)

    if not chunks:
        raise RuntimeError("pypdf visitor yielded no text chunks")

    # Split chunks into individual words; estimate bbox from font size
    words = []
    for (text, x_bl, y_bl, sz) in chunks:
        # Split on whitespace, estimate each word's x offset
        # pypdf gives us the start-of-chunk x; advance crudely by char_width ~ 0.5*sz
        char_w = sz * 0.5
        token_x = x_bl
        for token in re.split(r'(\s+)', text):
            if not token.strip():
                token_x += len(token) * char_w * 0.4
                continue
            w_width = len(token) * char_w
            x1 = token_x
            x2 = token_x + w_width
            # PDF y_bl is baseline (bottom-left); top of glyph ~ y_bl + sz*0.8
            y_top_pdf = y_bl + sz * 0.8
            y_bot_pdf = y_bl
            # Flip to top-left origin for match_span / write_highlight consistency
            y1_top = H - y_top_pdf
            y2_top = H - y_bot_pdf
            words.append((token, x1, y1_top, x2, y2_top))
            token_x += w_width

    if not words:
        raise RuntimeError("pypdf visitor: no word tokens after splitting")
    return words, W, H


# ---------------------------------------------------------------------------
# Tier 4: pdfminer.six
# ---------------------------------------------------------------------------

def tier4_pdfminer(pdf_path, page):
    """Returns (words, W, H) with top-left origin coords, or raises RuntimeError."""
    from pdfminer.high_level import extract_pages
    from pdfminer.layout import LTTextBox, LTTextLine, LTChar, LTAnno, LTPage

    words = []
    W, H = None, None

    for page_num, page_layout in enumerate(extract_pages(pdf_path), 1):
        if page_num != page:
            continue
        W = page_layout.width
        H = page_layout.height

        for element in page_layout:
            if not isinstance(element, LTTextBox):
                continue
            for line in element:
                if not isinstance(line, LTTextLine):
                    continue
                # Accumulate characters into word tokens
                current_word = []
                current_chars = []  # (x0, y0, x1, y1) per char in pdfminer coords (bottom-left)

                def flush_word():
                    if not current_word:
                        return
                    text = ''.join(current_word)
                    if text.strip():
                        x1 = min(c[0] for c in current_chars)
                        y1_pdf = min(c[1] for c in current_chars)
                        x2 = max(c[2] for c in current_chars)
                        y2_pdf = max(c[3] for c in current_chars)
                        # pdfminer uses bottom-left origin; flip to top-left
                        y1_top = H - y2_pdf
                        y2_top = H - y1_pdf
                        words.append((text, x1, y1_top, x2, y2_top))
                    current_word.clear()
                    current_chars.clear()

                for char in line:
                    if isinstance(char, LTChar):
                        if char.get_text() in (' ', '\t', '\n'):
                            flush_word()
                        else:
                            current_word.append(char.get_text())
                            current_chars.append((char.x0, char.y0, char.x1, char.y1))
                    elif isinstance(char, LTAnno):
                        if char.get_text() in (' ', '\t', '\n'):
                            flush_word()
                flush_word()
        break

    if W is None:
        raise RuntimeError(f"pdfminer: page {page} not found in document")
    if not words:
        raise RuntimeError("pdfminer: no words extracted")
    return words, W, H


# ---------------------------------------------------------------------------
# Tier 5: visual render via pdftoppm
# Renders page to PNG, reads it with the Read tool (caller handles this).
# Returns estimated coords from visual inspection - we do our best here.
# Since this runs as a script, we emit a special VISUAL_FALLBACK line and
# let the caller (citation-hunt skill) handle the image read step.
# ---------------------------------------------------------------------------

def tier5_visual_render(pdf_path, page):
    """
    Render page to PNG and print VISUAL_FALLBACK with the image path.
    The caller (citation-hunt) reads the image, visually locates the passage,
    and estimates fractional coords manually.
    Raises RuntimeError if pdftoppm is unavailable.
    """
    pdftoppm = subprocess.run(['which', 'pdftoppm'], capture_output=True, text=True)
    if pdftoppm.returncode != 0:
        raise RuntimeError("pdftoppm not available for visual fallback")

    tmp_dir = tempfile.mkdtemp(prefix='bbox_visual_')
    out_prefix = os.path.join(tmp_dir, 'page')
    result = subprocess.run(
        ['pdftoppm', '-f', str(page), '-l', str(page), '-r', '150', '-png', pdf_path, out_prefix],
        capture_output=True, text=True
    )
    if result.returncode != 0:
        raise RuntimeError(f"pdftoppm failed: {result.stderr.strip()}")

    # Find the generated PNG
    pngs = sorted(f for f in os.listdir(tmp_dir) if f.endswith('.png'))
    if not pngs:
        raise RuntimeError("pdftoppm produced no PNG output")

    img_path = os.path.join(tmp_dir, pngs[0])
    # Signal to caller: they should read this image and estimate coords
    print(f"VISUAL_FALLBACK: {img_path}", file=sys.stderr)
    raise RuntimeError(f"VISUAL_FALLBACK:{img_path}")


# ---------------------------------------------------------------------------
# Write highlight (unchanged from original; coords are top-left origin)
# ---------------------------------------------------------------------------

def write_highlight(pdf_path, page, word_span, W, H, color_rgb):
    """Write highlight annotation into PDF using per-word QuadPoints."""
    from pypdf import PdfReader, PdfWriter
    from pypdf.generic import (
        ArrayObject, DictionaryObject, FloatObject, NameObject,
        NumberObject, ByteStringObject,
    )
    import random, string

    reader = PdfReader(pdf_path)
    writer = PdfWriter()
    writer.append(reader)

    page_obj = writer.pages[page - 1]
    media_box = page_obj.mediabox
    page_h = float(media_box.height)

    quad_points = []
    all_x1, all_y1_pdf, all_x2, all_y2_pdf = [], [], [], []

    for (_, x1, y1_top, x2, y2_top) in word_span:
        # Flip from top-left to PDF bottom-left
        y1_pdf = page_h - y2_top
        y2_pdf = page_h - y1_top
        quad_points += [x1, y1_pdf, x2, y1_pdf, x1, y2_pdf, x2, y2_pdf]
        all_x1.append(x1); all_x2.append(x2)
        all_y1_pdf.append(y1_pdf); all_y2_pdf.append(y2_pdf)

    rect = [min(all_x1), min(all_y1_pdf), max(all_x2), max(all_y2_pdf)]

    r, g, b = color_rgb
    annot = DictionaryObject({
        NameObject('/Type'):       NameObject('/Annot'),
        NameObject('/Subtype'):    NameObject('/Highlight'),
        NameObject('/Rect'):       ArrayObject([FloatObject(v) for v in rect]),
        NameObject('/QuadPoints'): ArrayObject([FloatObject(v) for v in quad_points]),
        NameObject('/C'):          ArrayObject([FloatObject(r), FloatObject(g), FloatObject(b)]),
        NameObject('/CA'):         FloatObject(0.5),
        NameObject('/F'):          NumberObject(4),
        NameObject('/NM'):         ByteStringObject(
            ''.join(random.choices(string.ascii_letters + string.digits, k=8)).encode()
        ),
    })

    annot_ref = writer._add_object(annot)
    if '/Annots' in page_obj:
        page_obj['/Annots'].append(annot_ref)
    else:
        page_obj[NameObject('/Annots')] = ArrayObject([annot_ref])

    with open(pdf_path, 'wb') as f:
        writer.write(f)


# ---------------------------------------------------------------------------
# Main: run the fallback ladder
# ---------------------------------------------------------------------------

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--pdf',   required=True)
    parser.add_argument('--page',  type=int, required=True)
    parser.add_argument('--quote', required=True)
    parser.add_argument('--color', default='green', choices=COLORS.keys())
    args = parser.parse_args()

    pdf_path = os.path.expanduser(args.pdf)

    tiers = [
        ('pdftotext+xml',  tier1_pdftotext),
        ('pypdf-visitor',  tier3_pypdf),
        ('pdfminer',       tier4_pdfminer),
    ]

    words, W, H = None, None, None
    tier_used = None
    last_errors = []

    for name, fn in tiers:
        try:
            words, W, H = fn(pdf_path, args.page)
            tier_used = name
            print(f"TIER: {name}", file=sys.stderr)
            break
        except Exception as e:
            last_errors.append(f"{name}: {e}")
            print(f"TIER_FAIL: {name}: {e}", file=sys.stderr)

    # Tier 5: visual render - only reached if all text-layer tiers failed
    if words is None:
        try:
            tier5_visual_render(pdf_path, args.page)
        except RuntimeError as e:
            err = str(e)
            if err.startswith('VISUAL_FALLBACK:'):
                img_path = err.split(':', 1)[1]
                # Emit the special signal - citation-hunt will read the image
                # and estimate coords visually, then call write_highlight directly
                print(f"VISUAL_FALLBACK: {img_path}")
                sys.exit(2)  # exit code 2 = visual fallback needed
            last_errors.append(f"visual: {e}")

        print("MATCH_FAILED: all extraction tiers exhausted", file=sys.stderr)
        for e in last_errors:
            print(f"  {e}", file=sys.stderr)
        sys.exit(1)

    # Match the quote
    try:
        span = match_span(words, args.quote)
    except ValueError as e:
        print(f"MATCH_FAILED: {e}", file=sys.stderr)
        print(f"TIER_USED: {tier_used}", file=sys.stderr)
        sys.exit(1)

    # Fractional coords (top-left origin for org-noter)
    x1 = min(w[1] for w in span) / W
    y1 = min(w[2] for w in span) / H
    x2 = max(w[3] for w in span) / W
    y2 = max(w[4] for w in span) / H

    print(f"NOTER_PAGE: ({args.page} {x1:.10f} . {y1:.10f})")
    print(f"HIGHLIGHT: #s(pdf-highlight {args.page} (({y1:.10f} {x1:.10f} {y2:.10f} {x2:.10f})))")

    write_highlight(pdf_path, args.page, span, W, H, COLORS[args.color])
    print(f"WROTE: highlight on page {args.page} of {pdf_path}", file=sys.stderr)


if __name__ == '__main__':
    main()
interface:
  display_name: "Citation Hunt"
  short_description: "Find citations for the uncited claims in a section of text"
  default_prompt: "Use $citation-hunt to find citations for the uncited claims in this section."
{{ .chezmoi.homeDir }}/.agents/skills/citation-hunt

git

aliases

oh-my-zsh git plugin comes with many aliases:
run alias to view all
also handy to alias | grep rebase

git-absorb

git-absorb (opens in a new tab) automatically creates fixup commits for local changes and assigns
them to the commits they amend. It is useful for cleaning up a branch before
review or merge.

brew "git-absorb"

GPG

Examplebrew install gnupg
gpg --full-generate-key
gpg (GnuPG) 2.4.8; Copyright (C) 2025 g10 Code GmbH
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
   (9) ECC (sign and encrypt) *default*
  (10) ECC (sign only)
  (14) Existing key from card
Your selection? 1
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (3072) 4096
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y

GnuPG needs to construct a user ID to identify your key.

Real name: Anton Bilbaeno
Email address: antoaenono@pm.me
Comment:
You selected this USER-ID:
    "Anton Bilbaeno <antoaenono@pm.me>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? O
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: /Users/antoaenono/.gnupg/trustdb.gpg: trustdb created
gpg: directory '/Users/antoaenono/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/Users/antoaenono/.gnupg/openpgp-revocs.d/393A*.rev'
public and secret key created and signed.

pub   rsa4096 2025-10-19 [SC]
      393A*
uid                      Anton Bilbaeno <antoaenono@pm.me>
sub   rsa4096 2025-10-19 [E]

Then C-x C-f ./authinfo.gpg, write.
Trying to access again from Emacs fails, bcuz gpg subprocess isn't connected.

Examplebrew install pinentry-mac

Add to ~/.gnupg/gpg-agent.conf

Examplepinentry-program /opt/homebrew/bin/pinentry-mac

Make sure agent picks up config

Examplegpg-connect-agent reloadagent /bye

Now when accessing ./authinfo.gpg, dialog should appear for passphrase.

Notes

1

Donald Knuth introduced literate programming in 1984.

2

The Org document remains the source of truth.

3

Keep prose and code together in config.org.

Site settings

Reading progress
Reading progress