Skip to content

Retry setupDatabase writes that conflict with replication - #39

Open
peachbits wants to merge 1 commit into
masterfrom
matthew/setup-replication-conflicts
Open

peachbits wants to merge 1 commit into
masterfrom
matthew/setup-replication-conflicts

Conversation

@peachbits

Copy link
Copy Markdown
Contributor

CHANGELOG

Does this branch warrant an entry to the CHANGELOG?

  • Yes
  • No

Dependencies

none

Description

setupDatabase reads each setup document, compares it with the setup, and writes it if it is missing or different. If replication delivers the same document to that cluster between the read and the write, CouchDB rejects the write with 409 conflict ("Document update conflict."), and the error escapes setupDatabase.

This happens when a server writes a new design document to several clusters in turn and some of those clusters replicate into a later one. edge-login-server exits when setup fails, so the first backend restarted after a release that added a design document crash-looped. Each restart got one database further before the next conflict, until every database on the replicated cluster had the document. Any release that adds or changes a design document can hit the same race.

Both write loops in doSetup now run their read, check and write through a new retryOnConflict helper. On a 409, the step runs again with a fresh read:

  • A document that now matches (matchJson) needs no write, so setup moves on. This is the usual case, because replication delivers the same content setup was about to write.
  • A template that now exists is left alone, the same as when it exists at the first read. Setup still never overwrites an existing template.
  • Otherwise the step writes again on top of the new _rev.

After 5 conflicting writes for one document, setup throws the last conflict error. Other errors are thrown right away, as before.

The log lines are unchanged. "Writing document ..." is printed before each write, so a document that needs a second write logs it twice. A dry run never writes, so its output is the same as before.

Testing:

  • New test/couchdb/setup-database.test.ts, with a fake cluster that rejects stale writes with a 409 the way CouchDB does. Four cases fail on master with "Document update conflict." and pass on this branch. Two more check that other errors are not retried and that a dry run only logs.
  • Two throwaway CouchDB 3.5.2 instances, B continuously replicating from A. Each round, setup wrote a new design document version to A and then B, with B's write held back until replication had delivered A's copy. On master, setup threw 409 conflict in every round. On this branch, all 4 rounds hit the real 409 and finished, and both clusters ended on the same revision with no conflicts.
  • yarn verify passes.

Replication can deliver a setup document between the read and the
write in doSetup. CouchDB then rejects the write with a 409 conflict,
and setupDatabase threw. A server that exits on setup errors would
restart in a loop until every replicated cluster had the document.

On a conflict, run the same read-and-check step again. A document that
now matches, or a template that now exists, needs no write. Otherwise
write again on top of the new revision, up to 5 attempts in all. Other
errors still throw right away.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant