---
module: technical.journal
title: Journal Service — ATM journal storage, parsing & gap detection
tab: Overview
order: 10
audience: dev
---

> **DRAFT · internal.** Written 2026-07-01 from hiveops-journal source. Verify against code before relying on specifics.

## Responsibility

`hiveops-journal` receives raw ATM journal files from agents (chunked upload), stores the originals on disk, and parses them into structured transactions using YAML rules (VC journal format + TCR rules). It also detects missing journal dates per ATM (gaps), archives old transactions (90-day hot/cold split), and exposes a REST API consumed by the incident/devices frontends. It replaces the legacy SmartJournal dependency. See [platform.service-ownership].

Runs as **one image, two roles** driven by `JOURNAL_ROLE`:
- `upload` (default) — HTTP controllers, scheduled jobs, Kafka **producer**.
- `worker` — parse listener only; one worker container **per institution** (`JOURNAL_INSTITUTION`), Kafka **consumer**.

Server port **8087** (`server.port`, context-path `/`).

## Data

- **Database:** `hiveiq_journal` — connection via full JDBC URL in env var **`DB_URL`** (`spring.datasource.url=${DB_URL:...}`), not split host/port/name. `ddl-auto=none`; schema managed by **Flyway** (migrations `V1`–`V15`).
- **Secondary read-only datasource:** incident DB via `INCIDENT_DB_URL` — used to sync ATM→institution mapping.

Key entities (`com.hiveops.journal.entity`):

| Entity | Table | Purpose |
|--------|-------|---------|
| `JournalFile` | `journal_files` | Uploaded file metadata; `parseStatus` = PENDING/PARSING/PARSED/ERROR |
| `JournalTransaction` | `journal_transactions` | Parsed transactions (hot, < 90 days) |
| `JournalTransactionArchive` | `journal_transactions_archive` | Cold archive (no FK to files) |
| `JournalGap` | `journal_gaps` | Detected missing journal dates per ATM |
| `AtmInstitutionMap` | `atm_institution_map` | ATM→institution routing map (V14) |
| `ChequeImage` | cheque image table (V7/V8) | Cheque/deposit images from agents |

Notable migrations: `V10` archive table, `V14` ATM institution map, `V15` parse dedup fields. See [platform.data-stores].

## Kafka

Topic constants defined in `kafka/JournalKafkaConfig.java`. See [platform.kafka].

**Produced** (upload role):

| Topic | Producer class |
|-------|----------------|
| `journal.parse-requests.{INSTITUTION}` (prefix `journal.parse-requests.`, or `...UNASSIGNED`) | `JournalParsingService` (routes per institution) |
| `journal.replenishments` | `ReplenishmentEventProducer` |
| `journal.file-parsed` | `FileParsedEventProducer` |
| `hiveops.journal.transactions.recorded` | `JournalTransactionEventProducer` |

Static topic `journal.parse-requests` is also declared. Per-institution topics are created at startup from `JOURNAL_INSTITUTIONS` env plus an always-present `...UNASSIGNED`.

**Consumed:**

| Topic | Listener class | groupId |
|-------|----------------|---------|
| `journal.parse-requests.{INSTITUTION}` (SpEL `#{@parseTopicName}`) | `JournalParseConsumer` (worker role) | `#{@parseGroupId}` (derived from `JOURNAL_INSTITUTION`) |
| `journal.file-parsed` | `JournalGapDetectionService` | `journal-gap-detection` |
| `journal.file-parsed` | `CassetteInventoryService` | `journal-cassette-inventory` (offset reset = latest) |

## API surface

Agent-facing (no JWT):
- `GET /atm/journal/query` — resume offset for a file (legacy `FileUploadClient` protocol) — `AgentUploadController`
- `POST /atm/journal/upload` — chunked gzip upload of a journal file — `AgentUploadController`
- `GET|POST /api/agent/journal/query|upload` — newer agent journal path — `AgentJournalApiController`
- `POST /api/agent/image/upload`, `POST /api/agent/image/cheque/metadata` — cheque images — `AgentImageController`

Protected (`/api/journal`, `JournalApiController`, `@PreAuthorize("isAuthenticated()")`):
- `GET /api/journal/transactions` — list transactions (atmId, from, to, type, page, size)
- `GET /api/journal/transactions/{id}` — single transaction
- `GET /api/journal/transactions/summary` — aggregate summary
- `GET /api/journal/transactions/top-atms` — top ATMs by volume
- `PATCH /api/journal/transactions/{id}/adoons-analysis` — store AI (Adoons) analysis
- `GET /api/journal/files` — list uploaded files
- `GET /api/journal/files/{id}/raw` — stream raw file content (handles `.gz` transparently)
- `POST /api/journal/files/{id}/reparse` — re-queue file for parsing
- `GET /api/journal/files/download-zip` — bulk file download
- `GET /api/journal/gaps` — list detected gaps
- `PUT /api/journal/gaps/{id}/request-fetch` — mark gap FETCH_REQUESTED
- `GET /api/journal/cassette-config` — cassette configuration

Admin (`/api/journal/admin`, `AdminController`, all `@PreAuthorize("hasAnyRole('MSP_ADMIN', 'BCOS_ADMIN')")`):
- `POST /reload-rules`, `GET|PUT /rules`, `GET|PUT /rules/structured`, `GET /types`
- `POST /reparse-all`, `POST /parse-sync`, `POST /archive-now`
- `POST /reconcile` — re-queue stranded PENDING/PARSING files (**run after every deploy**)
- `POST /migrate-storage`, `POST /sync-cassette-inventory`
- `GET|PUT /tcr-rules`, `POST /tcr-rules/reload`, `GET|PUT /tcr-rules/raw`

Also: `/api/journal/cheques/...` (`JournalImageQueryController`), `/api/simulator/transactions` (`SimulatorController`).

## Gotchas

- **`DB_URL` is a full JDBC string** — unlike sibling services that use split `DB_HOST/DB_PORT/DB_NAME`.
- **Admin endpoints require `hasAnyRole('MSP_ADMIN', 'BCOS_ADMIN')`**; read/query endpoints only require `isAuthenticated()`; agent upload routes require **no JWT** (agent-facing).
- **Post-deploy reconcile is mandatory** — `POST /api/journal/admin/reconcile` re-queues files stranded in PENDING/PARSING when the instance restarted; skipping it leaves files stuck forever. A scheduled recovery also re-queues files stuck in PARSING > 30 min (`JournalParsingService`).
- **Per-institution routing** — parse requests go to `journal.parse-requests.{INSTITUTION}` (uppercase); unknown/unmapped ATMs fall back to `...UNASSIGNED` so no data is lost before a dedicated worker exists.
- **Dedup happens at two layers**: upload-time guard skips files already `PENDING`/`PARSING` in flight (`JournalUploadService`); parse-time `deduplicateSubTransactions` merges two-phase records that share a SEQ# (`JournalParser`).
- **SEQ# `0` is not a real host sequence** (deposits/settlements/no-card) — parser stores it as null, frontend renders `-`.
- **Compression:** agent uploads gzip → stored as `.txt`; after `journal.archive.days` (90) archival re-compresses to `.txt.gz` and updates `storage_path`. Reader/raw-view/reparse handle `.gz` transparently.
- **Workers use a smaller Hikari pool** (`JOURNAL_DB_POOL_SIZE`, default 30 for upload; workers set to ~5). Kafka producer fails fast (`max.block.ms=3000`) to avoid blocking request threads.
