docs(api): add API conventions, migrations, and README

[jacomago]

Document API conventions including RSQL query syntax, bulk operations, jobs/batch definitions, and migration notes under dev-docs/api/.

[anderslindho]

Overall the v1 direction here is solid: plural nouns, correct method semantics, RFC 9457 errors, split request/response schemas, jobs for async work, RSQL replacing the old boolean-DSL magic chars. All good moves, agree with merging that as the baseline.

Where I think this needs another pass is the batch/bulk section, specifically re how it's meant to actually serve RecCeiver.

batch vs reconcile

:batch's mixed post/put/patch/delete shape still needs the producer to know what already exists in CF (to bucket into post vs put) and what to delete (no query-scoped delete, no server-side reconcile, per the doc's own wording). That's the same read-before-write pattern we're trying to get RecCeiver away from. The doc even says it outright: "a producer that does not track what it holds builds the operations by diffing its desired set against a lean GET /channels...". If RecCeiver still has to GET-then-diff to build the batch, that's the same read, just done before submitting instead of after.

Proposal: add an actual reconcile op. Producer submits "this is my desired/observed set for scope X, generation N," CF upserts and tombstones anything in-scope not mentioned. That's the real one-way primitive. Once that exists I think the mixed-method :batch shape can go away entirely, reconcile covers what it was trying to do for RecCeiver, and I don't think anyone else needs mixed-method batch (open Q below).

Reconcile doesn't get to skip jobs/per-item results just because it's declarative. Still large-N (thousands of channels per IOC per our own README numbers), still partially-failable, and a failed item must not get tombstoned (it didn't disappear, it just failed validation), so reconcile needs per-item results plus sync-under-threshold/async-over-threshold, same machinery batch has today, just wrapped around a "desired state" body instead of an op-list.

generic (non-reconcile) bulk, leaving this open

Once reconcile absorbs RecCeiver: is there still a real need for generic bulk create/update/delete (admin tooling, migration scripts, whatever)? If yes, and it's small-N/interactive (our own actor table already has admin/curator as "occasional, deliberate"), I don't think that needs jobs or per-item results at all. Validate-then-write, atomic, single response and status. Simpler for callers than today's 200-with-embedded-422s. If there's actually a consumer doing large/failure-prone homogeneous bulk writes outside RecCeiver, that path needs per-item handling too, same as reconcile, just without tombstoning. Need to know if that consumer exists before deciding.

200 vs 207 vs job, the doc contradicts itself here

Hard rules say "never 200 with an error body." Async section says "never... a 207 Multi-Status." Sync :batch's own example is a 200 with embedded per-item 422s. All three can't be true at once. If the intent is ":batch is a command, 200 = command ran, per-item statuses are data not errors," fine, but that needs to be written as an explicit carve-out to the hard rule, not left implicit.

"mirrors ES _bulk", ground this in domain not backend

Non-atomicity and the 200-status choice for batch are both justified by "follows Elasticsearch _bulk," but the doc is explicit elsewhere (opaque cursor) that engine behavior shouldn't leak into the contract. Same conclusion probably still holds, just ground it properly: large batches need forward progress, one bad item shouldn't sink 5k good ones, and no backend gives free cross-doc atomicity at this scale anyway. That reasoning would survive a hypothetical backend swap. The ES justification wouldn't.

native concepts needed for reconcile

For tombstoning to be safe I think we need, minimum:

• an opaque source_id/scope key, producer-supplied, CF doesn't interpret it (same idea as the opaque pagination cursor already in here)
• a generation/sequence guard per scope, so a delayed or reordered reconcile submission can't tombstone state a newer one already established
• first-class lifecycle fields on the channel (state, last_seen_at, tombstoned_at) instead of inferring lifecycle from arbitrary metadata, makes purge policy simpler to query, lets GET /channels default-exclude tombstoned channels, and gives an audit trail without extra work

Don't think we need IOC / RecCeiver / session to become their own CF resources though, scope just needs to be whatever unit appears/disappears together. RecCeiver owns getting that scoping right. CF just needs the opaque key plus generation guard.

one more thing this forces us to decide now, not later: can one channel have multiple hosts?

Separately from all of the above, PVs can be hosted by more than one IOC at once. Today CF pretends this can't happen, so when more than one recceiver/IOC reports the same name, whichever write lands last just silently clobbers the previous metadata. No visibility that it happened.

This lands directly on the source_id-scoped reconcile design above, not just future work. As sketched, if source A and source B both claim channel X, B's reconcile submission would silently overwrite A's metadata for X, and A's next reconcile would silently overwrite it back. Same bug we're trying to kill, just now inside the new primitive instead of the old one. So reconcile can't be specced without deciding: is an upsert a per-source claim on a channel, or a global clobber?

I think existence and metadata split cleanly here. Existence can aggregate now: channel exists if ≥1 source claims it, tombstone only when the last claim drops, that's just reference counting, solvable today. Metadata conflict is harder: two sources reporting different properties/tags for the same name doesn't aggregate cleanly, and I don't think we need to solve that now (whether we warn, reject, or just surface both isn't decided). But the identity question, keyed by name alone vs. some (name, ioc_id)-style composite, exact keying TBD, needs to at least be flagged as open now, because retrofitting single-owner -> multi-claim later is a lot more expensive than leaving room for it in the reconcile primitive from the start.

[jacomago]

batch vs reconcile

I can swap to reconcile.

native concepts needed for reconcile

For tombstoning to be safe I think we need, minimum:

I still don't understand tombstoning, is this a feature inside of channelfinder? Or is this a client is pushing some extra metadata?

Can you give an example of using just the normal update a single channel example?

one more thing this forces us to decide now, not later: can one channel have multiple hosts?

urgl, I don't particularly want to tackle this as it requires doing an elastic migration. The channel name is the key in the channels 'table'. Adding a source_id (which I think would have to be recceiver:ioc_id) Would mean adding a migration to elastic which we've never done before. The support for migrations also kind of ... is not nearly as good as flyway is for postgres.

Which completely changes the scope of this PR. Adding support for a source_id, which I think is pretty easily added on top instead, we modify the Channel Response and Request object with an optional field source_id. This would not break a v1 api. I think it does not need to be added now for this reason.

[simon-ess]

There is some ambiguity here: What if I have a property named tag?

[jacomago]

Nice catch.

[simon-ess]

I still don't understand tombstoning, is this a feature inside of channelfinder? Or is this a client is pushing some extra metadata?

From what I read (e.g. https://en.wikipedia.org/wiki/Tombstone_(data_store)) it seems not obviously relevant in our case as we do not have a distributed database to deal with, but perhaps it is the correct approach regardless?

[anderslindho]

Well, my thought was that recceiver would be the one deleting channels, but this is really the crux between 'resolve' [sic] vs 'batch'. Does the recceiver calculate the sync or does channelfinder.

ChannelFinder - recCeiver always just pushes what it has. Else it needs state/db. Good that you're up for swapping to reconcile. @simon-ess's "why do we need to create, update, update, and delete all at once?" on the batch example is the same question from the other end I think.

I still don't understand tombstoning, is this a feature inside of channelfinder? Or is this a client is pushing some extra metadata?

Can you give an example of using just the normal update a single channel example?

(SR) From what I read (e.g. https://en.wikipedia.org/wiki/Tombstone_(data_store)) it seems not obviously relevant in our case as we do not have a distributed database to deal with, but perhaps it is the correct approach regardless?

TBH I am not 100% sure if it is the correct term, but I think it covers it. What I mean by it is that CF alone is responsible for changing channels from active to inactive. So my intention is basically soft delete + retention/purge policy, and the important part is who owns that (CF).

To answer the either/or directly: today it is a client pushing extra metadata. That's pvStatus. Client decides a channel went away, writes that decision into CF as a property. That's the pattern I want gone.

Example

Say SOME:AI stops being reported.

Today: recCeiver reads CF, works out that SOME:AI is no longer in its set, and writes the deactivation itself (pvStatus -> Inactive, or a delete). CF is a dumb store, the client holds the logic.

Proposed: recCeiver sends its current set for its scope, SOME:AI isn't in it, CF notices it's in-scope-but-absent and does the equivalent of PATCH /channels/SOME:AI {"state": "inactive"} on its own. Record stays, GET /channels excludes it by default, it comes back active on the next upsert that mentions it. recCeiver never reads and never computes a diff.

one more thing this forces us to decide now, not later: can one channel have multiple hosts?

urgl, I don't particularly want to tackle this as it requires doing an elastic migration. [...]

Which completely changes the scope of this PR. Adding support for a source_id, which I think is pretty easily added on top instead, we modify the Channel Response and Request object with an optional field source_id. This would not break a v1 api. I think it does not need to be added now for this reason.

I think we should separate API design from business logic design. What I think is important is that the API already now accommodates for this design. How exactly that is implemented and/or how dbs are migrated to support it is a separate issue.

Not asking for the migration, and definitely not in this PR. But "add it on top later" only holds for the field, not for the semantics. Schema-wise you're right, optional source_id is additive. Semantically no: whether an upsert is a per-source claim or a global clobber is observable behavior clients build on. Change it later and the same request does something different.

And it's not really future work, it's load-bearing for reconcile being correct. With a single source_id, a channel hosted by two IOCs gets deactivated by whichever one drops it first, even though the other still has it. Live PVs flapping in and out of GET /channels. The old failure was two writes clobbering each other; the new one is CF soft-deleting things on its own bad inference. Worse, because now it's our logic doing it.

Doesn't mean solving it now. But reconcile's rule needs to be "deactivate when nothing claims it" rather than "deactivate when the recorded source dropped it", and v1 can implement that with one claim per channel as a documented limitation. Free to write down now, expensive once clients have built on one-source-per-channel being the actual rule.

[shroffk]

one more thing this forces us to decide now, not later: can one channel have multiple hosts?

Separately from all of the above, PVs can be hosted by more than one IOC at once. Today CF pretends this can't happen, so when more than one recceiver/IOC reports the same name, whichever write lands last just silently clobbers the previous metadata. No visibility that it happened.

During the meeting we agreed that I will be adding a "preprocessor" SPI similar to the "processor" one that exists.

So each site can add their own validation for channels... one of these can be a validator that detects if existing/active channels are there in channelfinder which have different hosts, it can then log or notify the appropriate entities to resolve the issue.

I don't think we should allow multiple properties with the same name.

[tynanford]

Overall I really like this. RSQL queries would be great.

I'm also thinking more about the duplicate PV names. I agree with @anderslindho that it is good to think about even if we don't address/support it fully in v1. I'm not saying this is practical but if PV name is the key and a PV can have multiple iocid, hostname, etc., then a channel in CF could be represented like this:

{
    "name": "SR12C:SQSF1:Setpoint",
    "properties": {
      "Acc": "SR",
      "Family": "SQSF",
      "Field": "Setpoint",
      "Sector": "12",
      "Device": "1",
      "Position": "187.277343",
      "hostName": "ioc-host-a.example.local",
      "iocName": "skewquads2",
      "iocid": "192.168.20.11:45334",
      "iocIP": "192.168.20.11",
      "time": "2026-07-15 09:12:00.123456",
      "caPort": "45502",
      "recordType": "ao",
      "recordDesc": "Current setpoint",
      "archive": "Slow",
      "Engineer": "jdoe",
      "Location": "SoftIOC"
    },
    "pvStatus": "Active",
    "conflict": {
      "duplicateActiveSources": true,
      "activeSourceIds": [
        "recceiver-1.example.local:192.168.20.11:45334",
        "recceiver-2.example.local:192.168.20.12:52110"
      ],
      "note": "flattened properties above reflect one of these sources arbitrarily; do not treat as authoritative until resolved"
    },
    "tags": ["MML", "PS"],
    "sources": [
      {
        "source_id": "recceiver-1.example.local:192.168.20.11:45334",
        "pvStatus": "Active",
        "last_seen": "2026-07-15T09:12:00Z",
        "properties": {
          "recceiverID": "recceiver-1.example.local",
          "iocName": "skewquads2",
          "hostName": "ioc-host-a.example.local",
          "iocid": "192.168.20.11:45334",
          "iocIP": "192.168.20.11",
          "time": "2026-07-15 09:12:00.123456",
          "caPort": "45502",
          "recordType": "ao",
          "recordDesc": "Current setpoint",
          "archive": "Slow",
          "Engineer": "jdoe",
          "Location": "SoftIOC"
        }
      },
      {
        "source_id": "recceiver-2.example.local:192.168.20.12:52110",
        "pvStatus": "Active",
        "last_seen": "2026-07-15T09:10:45Z",
        "properties": {
          "recceiverID": "recceiver-2.example.local",
          "iocName": "skewquads2-test",
          "hostName": "ioc-host-b.example.local",
          "iocid": "192.168.20.12:52110",
          "iocIP": "192.168.20.12",
          "time": "2026-07-15 09:10:45.987654",
          "caPort": "45519",
          "recordType": "ao",
          "recordDesc": "Current setpoint (test IOC)",
          "archive": "Slow",
          "Engineer": "jdoe",
          "Location": "SoftIOC"
        }
      },
      {
        "source_id": "recceiver-1.example.local:192.168.20.13:38821",
        "pvStatus": "Inactive",
        "last_seen": "2026-06-20T14:22:03Z",
        "properties": {
          "recceiverID": "recceiver-1.example.local",
          "iocName": "skewquads2",
          "hostName": "ioc-host-old.example.local",
          "iocid": "192.168.20.13:38821",
          "iocIP": "192.168.20.13",
          "time": "2026-06-20 14:22:03.456789",
          "caPort": "45506",
          "recordType": "ao",
          "recordDesc": "Current setpoint",
          "archive": "Slow",
          "Engineer": "jdoe",
          "Location": "SoftIOC"
        }
      }
    ]
  }

Each source in this case is an IOC connection to a specific recceiver instance and includes all the properties that IOC reported. The top level properties is the flattened view for normal clients and includes properties manually added by non-recceiver clients. The top level pvStatus is an OR of all the source pvStatus fields and there is a conflict flag in case there are multiple active sources.

There are some problems with this idea (including an IOC restart adding a new source every time) but throwing it out there to brainstorm.

[tynanford]

oh that json blob reminded me of an old complaint about CF here: all the property values are strings so things like position in the example above are hard to sort on unless you make all positions the same number of characters. We added leading zeros in those cases. Not sure if it can be addressed or makes sense but did come up here

[jacomago]

Changes:

• Tried to simplify the documents
• Added sources, a more minimal version of what @tynanford added but with a lot of the same features
  • I went for a default of source_id: 'recceiver_id:ioc_name' as 'iocid' has problem that restarts gets new ports and hence new ioc_ids. But I also put its up to the user of the api to choose the source_id (i.e. the recceiver)
  • didn't track old property values, not sure how useful that is from just knowing there are multiple sources and then people can fix
• Swapped to reconcile rather than batch
  • Added generation for when two reconciles conflict
• Added design of a timeout on channels existing after inactive, same with sources
• dropped the owner
• basically wrote how lifecycle of a channel could be handled by ChannelFinder itself, tried to add a comment about how that could still work with v0 api.
• Add that property value can be float, integer or string, which i was thinking of adding before anyway.

[anderslindho]

Nice, this took on basically everything that I hoped for. Few things still open.

Biggest is source_id granularity. Mechanism is fine, but on the recommended recceiver_id:ioc_name: "IOC-name" is an ESS-ism, and, although it can be an example, it should not be the canonical form. And it might be useful to state the actual rule, which is that scope has to be the smallest thing that appears/disappears together. Same PV under multiple IOCs on one recCeiver only works if you scope per-IOC - then X has a claim from ioc-A and ioc-B and stays active till both drop. Scope per-recCeiver and one IOC dropping deactivates PVs the other still serves. So IMO state the granularity rule, and say same-PV-under-multiple-scopes is expected multi-claim, not an error.

Batch being gone is good, but it leaves no generic bulk primitive at all - bulk just means reconcile now. Anything non-recCeiver wanting high-volume writes (admin, migration) is stuck looping singles or faking a source_id. Completely fine for me, but maybe worth stating explicitly - your call.

Reconcile items reference properties/tags and an undefined one 422s per-item, so a recCeiver pushing a new property has to define it first or those items fail. That pre-provision step cuts against one-way. Should reconcile create missing definitions, or do producers pre-provision and we document it?

conflict:true is a good call. But: you can see two sources disagree but not what the losing value was (probably ok since we dropped per-source props but might warrant call-out), and doesn't say when conflict clears.

Minor: source_id optional with host fallback reintroduces the host instability we're avoiding, for whoever doesn't set it. Might be worth making required.

None of this blocks merge for me.

[jacomago]

Nice, this took on basically everything that I hoped for. Few things still open.

Biggest is source_id granularity. Mechanism is fine, but on the recommended recceiver_id:ioc_name: "IOC-name" is an ESS-ism, and, although it can be an example, it should not be the canonical form. And it might be useful to state the actual rule, which is that scope has to be the smallest thing that appears/disappears together. Same PV under multiple IOCs on one recCeiver only works if you scope per-IOC - then X has a claim from ioc-A and ioc-B and stays active till both drop. Scope per-recCeiver and one IOC dropping deactivates PVs the other still serves. So IMO state the granularity rule, and say same-PV-under-multiple-scopes is expected multi-claim, not an error.

That's why I made it recommended, its up to the producer to decide but something per-IOC is best. And something that survives reconnects of IOCs, which iocid does not. But I'll try make language clearer about what I mean by 'recommended'.

Reconcile items reference properties/tags and an undefined one 422s per-item, so a recCeiver pushing a new property has to define it first or those items fail. That pre-provision step cuts against one-way. Should reconcile create missing definitions, or do producers pre-provision and we document it?

I will add produces pre-provision. This is how recceiver works now, and since property list is small I don't see a big problem with checking properties exist or just pushing the properties at start.

conflict:true is a good call. But: you can see two sources disagree but not what the losing value was (probably ok since we dropped per-source props but might warrant call-out), and doesn't say when conflict clears.

I might just drop this, multiple sources always suggests a conflict of some kind. Could just be latest time seen.

Minor: source_id optional with host fallback reintroduces the host instability we're avoiding, for whoever doesn't set it. Might be worth making required.

Updated the fallback to be based on iocname and iocid property + the host

None of this blocks merge for me.

[tynanford]

At our site we sometimes struggle with the duplicate IOCs being started on the same host and recceiver_id:ioc_name won't catch that. Using iocid has the problem of the port but what about an optional feature that any inactive sources are removed when a new active source appears? Maybe one issue would be a race condition between an IOC restarting and recceiver/CF changing the old source_id to inactive. Then you have 2 active sources for a bit, which might be an issue for how to clean that up later.

You also then lose the ability to see the history of a PV moving from IOC-A to IOC-B. Hmm.

I'm not sure we would want the purge timeout here but I assume that can be disabled/set to 0. If a PV goes inactive because it was retired, we want that PV to stay in CF for a while in case people search for it.

[simon-ess]

At our site we sometimes struggle with the duplicate IOCs being started on the same host and recceiver_id:ioc_name won't catch that. Using iocid has the problem of the port but what about an optional feature that any inactive sources are removed when a new active source appears? Maybe one issue would be a race condition between an IOC restarting and recceiver/CF changing the old source_id to inactive. Then you have 2 active sources for a bit, which might be an issue for how to clean that up later.

I'm not sure I follow here. We have hosts with multiple IOCs, so how would you know which one to remove when a new one appears?

Would it be possible to have the reccasters send an optional iocid parameter which is used, can be site-specific, and defaults to the old iocid if not defined or if a collision occurs?

[simon-ess]

When will two producers share the same source ID? Does that mean that the recceivers are configured with the same ID and they both find an IOC with the same name/id?

[jacomago]

Yes exactly. Two recceivers on same subnet, you might want them as a redundant pair. One gets restarted while the other stays up, the source_id stays the same and conflicts are handled by the generation param.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[tynanford]

I'm not sure I follow here. We have hosts with multiple IOCs, so how would you know which one to remove when a new one appears?

The case I am thinking of is PV ABC exists in IOC XYZ which connects to recceiver-a. If XYZ IOC gets started twice on the same host/subnet, then each IOC instance will have the same source ID: recceiver-a:XYZ and CF won't catch the fact that there are actually two instances of PV ABC. Whereas using recceiverid:iocid would correctly show there are two sources here.

With iocid there is the issue of an IOC restart creating a new source_id. I think it is more honest though since iocid uniquely identifies the IOC/recceiver connection unlike just iocname/recceiverid

My question is can we make this workable, i.e. what about if a new active source appears, the inactive sources get purged automatically? Then IOC restarts don't continuously increase the number of sources.