---
module: transactions.journal-parsing
title: Journal Parsing Rules
tab: Architect
order: 30
audience: dev
---

How the Journal Parsing feature is actually built. The UI lives in the Transactions SPA, but the entire domain — rules, files, transactions, parsing — is owned by **hiveops-journal**. See [platform.service-ownership] and [platform.data-architecture].

## Services involved

- **hiveops-transactions** — the Svelte SPA (`JournalParsing.svelte`, `JournalReparse.svelte`) plus a thin stateless proxy. It owns no data. Its `JournalProxyController` forwards any `/api/journal/**` request verbatim to the journal service (`services.journal.url`, default `http://hiveiq-journal:8087`). Only `Authorization` + `Content-Type` headers survive the proxy. See [technical.transactions].
- **hiveops-journal** — owns rules, files, transactions, parsing, and Kafka. Split into two runtime roles from one image via `journal.role`:
  - **upload role** (`journal.role=upload`, default) — runs HTTP controllers (`AdminController`, `JournalApiController`), scheduled jobs, and the Kafka **producer**.
  - **worker role** (`journal.role=worker`) — runs only `JournalParseConsumer` (the Kafka **consumer**) + actuator. `AdminController` is `@ConditionalOnProperty(journal.role=upload)`, so admin endpoints exist only on the upload container.

## Data flow — rules edit

1. `JournalParsing.svelte` → `journalAdminAPI.getRules()` → `GET /api/journal/admin/rules` (text/plain YAML).
2. Edit → `saveRules()` → `PUT /api/journal/admin/rules` (text/plain body) → `JournalRulesService.saveRulesContent()` writes `journal-rules.yml` and hot-reloads in place, returning the new `version`.
3. `reloadRules()` → `POST /api/journal/admin/reload-rules` re-reads the file without writing.
4. Vendor types → `GET /api/journal/admin/types` → `JournalRulesService.getAvailableTypes()` (names under `journal_types:`).
5. TCR variant uses a parallel `tcr-rules.yml` via `TcrRulesService` (`/api/journal/admin/tcr-rules[/raw|/reload]`).

No database writes for rules — they live in a YAML file on the journal container, loaded into an in-memory `JournalRules` model.

## Data flow — reparse (executor → Kafka → worker → record-store)

1. UI → `journalAdminAPI.reparseAll(atmId?, journalType?)` → `POST /api/journal/admin/reparse-all`.
2. `AdminController.reparseAll` selects `JournalFile` rows via `JournalFileRepository` (`findByAtmId` / `findByJournalType` / `...JournalTypeNot`, filtered by `NH_TCR` for the ATM vs TCR split).
3. For each file it: `transactionRepository.deleteByJournalFile(jf)` → sets `parseStatus=PENDING`, clears `transactionCount`/`parseError`/`lastParsedSize` → saves → calls `parsingService.parseJournalFileAsync(jf)`.
4. `JournalParsingService.parseJournalFileAsync` resolves the file's institution (persisting it if newly resolved) and **produces a Kafka message**: `kafkaTemplate.send(topic, atmId /*key*/, String.valueOf(fileId))`.
5. Topic is per-institution: `TOPIC_PARSE_PREFIX` = `journal.parse-requests.` + `INSTITUTION` (uppercase), or `journal.parse-requests.UNASSIGNED` when the institution isn't in configured `journal.institutions`. See [platform.kafka].
6. `JournalParseConsumer` (`@KafkaListener(topics = "#{@parseTopicName}")`, worker role) receives the file id and calls `JournalParser.parseJournalFile(jf)`, which rebuilds `journal_transactions` rows from the original file on disk.

`reconcileStranded()` (upload role, on startup and via `POST /api/journal/admin/reconcile`) re-publishes PENDING/PARSING files that a restart left un-consumed.

## DB tables (hiveiq_journal)

- **`journal_files`** — read + written by reparse (status, counts, institution, `lastParsedSize`). Entity `JournalFile`, repo `JournalFileRepository`.
- **`journal_transactions`** — deleted then rebuilt on reparse; the parser writes rows here. Repo `JournalTransactionRepository` (`deleteByJournalFile`).
- Related (not edited by this screen): `journal_transactions_archive`, `journal_gaps`, `atm_institution_map`.

Transactions has **no database**; devices/ATM identity lives elsewhere. See [platform.data-architecture].

## Key API endpoints (all under the transactions `/api/journal/**` proxy → journal)

| Method | Path | Purpose |
|--------|------|---------|
| GET | `/api/journal/admin/rules` | Raw YAML rules (text/plain) |
| PUT | `/api/journal/admin/rules` | Save + hot-reload YAML |
| POST | `/api/journal/admin/reload-rules` | Reload rules from disk |
| GET | `/api/journal/admin/types` | Vendor/journal type names |
| GET/PUT | `/api/journal/admin/rules/structured` | Structured (JSON) rules |
| GET/PUT | `/api/journal/admin/tcr-rules` + `/tcr-rules/raw`, POST `/tcr-rules/reload` | TCR rules |
| POST | `/api/journal/admin/reparse-all?atmId=&journalType=` | Bulk reparse |
| GET | `/api/journal/files?atmId=&journalType=&page=&size=` | File parse status |

## Kafka topics

- Producer: upload role → `journal.parse-requests.<INSTITUTION>` (keyed by `atmId`), fallback `journal.parse-requests.UNASSIGNED`. Constant `JournalKafkaConfig.TOPIC_PARSE_PREFIX`.
- Consumer: worker role `JournalParseConsumer`, one worker per institution topic. See [platform.kafka].

## Roles / security

- Admin endpoints: `@PreAuthorize("hasAnyRole('MSP_ADMIN','BCOS_ADMIN')")`.
- `GET /api/journal/files`: `@PreAuthorize("isAuthenticated()")`, scoped by `InstitutionContext.resolveScope()` in the default branch.
- The transactions proxy gates `/api/**` broadly (`USER,CUSTOMER,ADMIN,MSP_ADMIN,QDS_ADMIN,BCOS_ADMIN`); the real authorization is journal's `@PreAuthorize`. See [platform.service-ownership].
