BP-46: Running without journal proposal

[Vanlightly]

Includes the BP-46 design proposal markdown document.

Master Issue: #2705

[eolivelli]

great work.
I had in my mind some ideas totally in this direction, so I am very happy to see this approach.

[eolivelli]

why this is related to this BP ?

[Vanlightly]

It's related. We're protecting against silent data loss that can cause inconsistency in the protocol. The cookies already played a role but we wanted the new mechanism to play well with the cookie mechanism and improve it.

[eolivelli]

are you talking about automatically rewriting the cookie in case of mismatch ?

[Vanlightly]

Yes, in case of mismatch, the cookie gets rewritten after all ledgers have been fenced and non-closed ledgers put in-limbo, as part of phase one, pre boot sequence. It allows the bookie to automatically handle its own recovery, rather than require operator intervention.

[ivankelly]

@Vanlightly we should have a flag for whether we automatic rewriting is allowed. We've been burned in the past by misconfiguration allowing bookies to come up where it really should have been kicked back to a human. Or maybe not a flag, but the steps to unjam yourself from a cookie mismatch should be a single command.

[Vanlightly]

Agreed. Allow through configuration for automatic cookie rewriting to be enabled or disabled. If the admin has it disabled and cookie validation does fail then as @ivankelly suggested, it would be good if the admin can run a command to resolve the cookie mismatch and execute the data repair.

[eolivelli]

@dlg99 I see your point. It makes sense.

+1 for adding a configuration option, disabled by default.

But, can't we keep this cookie part out of this BP ? This BP is about running without journal, not running on ephemeral disks

for the Cookie part we do not need a BP, you can simply send a PR, I imagine it will be an easy and simple patch

[Vanlightly]

@eolivelli I'd prefer to keep the cookie stuff in this BP as we have a working implementation ready to merge. Changes now would require refactoring and our main priority is syncing with OSS. The flag we will add of course but further changes, unless really necessary, we'd like to avoid.

[jvrao]

Yes, in case of mismatch, the cookie gets rewritten after all ledgers have been fenced and non-closed

How can you fence a non-closed ledger? Writer might have changed the ensemble and picked some other bookie to continue and may even come back to this bookie after another ensemble change. I may have to read down your exact proposal on how to detect and correct/fix.

running bookie with the data on ephemeral storage

Imagine a situation where Writer is writing to ephemeral storage with Qa=2 Qw=3 En=3 (B0, B1, B2). If a Bookie (B0) goes down and comes up with loss of data (ephemeral) and if that bookie fences can we make the writer continue to write the ledger as it could cotinue to receive 2 acks from other two bookies (B1, B2)?

[Vanlightly]

Regarding the fencing. The bookie only fences itself, this has no impact on any other bookies. Later on it performs recovery, which we know is correct no matter what any other writer is doing.

Regarding the ephemeral storage. Yes, another writer could still make progress, until the bookie initiates recovery and closes the ledger. This does not result in data loss.

[dlg99]

Should we keep the bookie as read-only until all ledgers are out of limbo to prevent slow bookies affecting the cluster? I assume that Phase Two creates additional IO and processes multiple ledgers in parallel.

[ivankelly]

There's two I/Os we need to concern ourselves here.

• The check for whether the bookie has the entries it should have. This check runs against the index, so should not impact other traffic.
• The copying of missing entries. In the common case, the number of entries to be copied should be a rounding error in terms of I/O. The only case where it would be significant is if the disks have been wiped and the bookie is trying to reconstruct the full contents. In this case, I agree it may make sense to make the bookie read-only.

[jvrao]

This check runs against the index,

Index file on the disk is also might not have flushed. right? Unless you are talking with ZK metadata comparison.

[ivankelly]

With DbLedgerStorage, the index is flushed directly after update, which happens in a batch as the entrylog is flushed.

[dlg99]

what happens if bookie runs with journalWriteData set to true, then journalWriteData set to false, bookie reboots and ledgers are in limbo state? What is the order of recovery then? Do we do the "limbo processing", if yes - is it happening before or after recovery from journal?

[Vanlightly]

Phase one is executed at the time of the cookie validation, which is pre boot. Phase two, currently named DataIntegrityService, is inserted into the boot sequence immediately after the AutoRecoveryService. This means that phase two if run, is run after the journal has been replayed.

[dlg99]

@jvrao You might be interested in this

[dlg99]

Did I understand it correctly that LAC behavior does not change except readLAC can get entryId that hasn't been fsynced to the disk yet?

What happens in case of coordinated restart of ES bookies (or the whole DC, fwiw)?

[Vanlightly]

If it is a controlled restart, then all bookies should be able to flush all their data to disk. So everything would be fine.

There is a risk that the graceful shutdown ends up not so graceful though, such as k8s not waiting long enough before killing the pods. Any early termination should be detected as an unclean shutdown.

If it were a DC power outage then we lose unflushed data across the cluster.

Either way, we have elevated risk of data loss. Any given ledger can recover from lossy writes and fencing ops without data loss only if there is the number of overlapping unclean shutdown and recoveries is < Ack Quorum bookies. Once we reach AQ, there could exist one or more entries that only reached AQ and were hosted on the affected bookies and lost, therefore being unrecoverable. We just need an entry to remain intact on a single bookie for it to be recoverable.

It would make the use of AZs even more important and to have good automation that does not kill bookies if they take a long time to shutdown.

[Vanlightly]

Regarding your question about readLac. A limbo ledger returns an EUNKNOWN as it cannot safely answer that question.

[Vanlightly]

Another topic is that of plugging gaps in a ledger that has experienced entry loss. If an entry is lost, then readers are blocked at that point. On our radar is a command to plug holes in the ledger with some kind of no op entry that the client skips. It allows for continued availability of the ledger, even though it experienced data loss. Such a mechanism is not included in this BP though.

[dlg99]

Currently readLac returns N and you can read entry N now and after DC-wide power loss.
With the changes you can read entry N now but unluckily timed after DC-wide power loss whole WQ will return EUNKNOWN.

I understand that this has to be expected in this case but I'd like to consider some additions:

• Consider having a way to get something like readLacPersisted() (or readLac(boolean isPersisted)) to have a way to distinguish potentially lossy from persisted tail entries
• and/or have journal bypass configurable on the per ledger handle level
• have ledger close() return after all ledger entries (at least AQ) are flushed to entry logs and fsynced.

Also document durability expectations for the tail entries with clear explanation of the risks to let people make more educated choice.

Otherwise the only option is to have two bookie clusters (strong durability and journal bypass) which adds operational overhead + more changes in the app.

The usecase is an app using BK for its WAL with strong durability requirements and for data where open to close durability matters but failure in the process is recoverable. i.e. lost entry at the tail of WAL may prevent app from recovering gracefully.

@jvrao fyi

[Vanlightly]

@dlg99 These additions are very interesting and this BP lays the foundation for such additional features. I recommend that we leave this BP as is and then once our implementation is merged we consider these potential additions as a next step along those lines.

[jvrao]

Did we consider ELPL to get-rid of journal? With ELPL it is kind of each ledger having its own consistency. With the addition of ledger level flags of durability levels with flush on close option that @dlg99 mentioned we can achieve all the current durability with ledger level granularity and completely get rid of journal.

[jvrao]

If it were a DC power outage

There are many k8s scenarios where the orderly shutdown may not happen. There are various scenarios but I guess we can improve ops process to make it happen. But that would take some effort.

[Vanlightly]

As far as I know from my research into BookKeeper design decisions, Entry Log Per Ledger (ELPL) can create problems with high volumes of active ledgers, so is not a universal solution. Perhaps @ivankelly can comment on that as he know more than me on that subject.

[ivankelly]

Did we consider ELPL to get-rid of journal?

We did not. ELPL requires random writes per ledger, which negatively impacts performance in the presence of many ledgers, which was not acceptable in our usecase.

[fpj]

Running without the entry log also halves the IOs, why keep the journal and not the entry log? It would be useful to explain as part of the motivation.

[jvrao]

Public clouds charge by the IOPS. This could be another argument we could make. Journal+Index+EntryLog; we have great scope to reduce the storage and IOPS requirements.

[fpj]

Writeset is the set of bookies that have acknowledged? Fragment maps to a contiguous range of entry ids?

[Vanlightly]

The writeset of an entry is the set of bookies that should host that entry, whose cardinality is the write quorum. A fragment is commonly referred to as an ensemble, though that word gets a little overloaded. Concretely a fragment is one kv entry in the ledger ensemble metadata, where the key is the first entry id of the fragment, and the value is the ensemble of bookies responsible for that range. The range is bounded by the key (first entry) and either the next fragment (exclusive) or the end of the ledger (inclusive). That range is contiguous.

[fpj]

Right, that's a concept that's in the code, makes sense. We might want to reflect some of that in Javadocs, but that's a separate issue.

[fpj]

Given that the data is not being synced to disk on acknowledgment, there is no guarantee that a single copy will make it to enable repair, is it right? There is only a weak promise that any given entry is repairable in the case that some bookie was able to get it to disk before crashing.

[Vanlightly]

For an acknowledged entry to be lost it, every single bookie that received the entry (write quorum in the normal case, ack quorum in some edge cases) would need to simultaneously be terminated before it could flush to disk. This gives us Apache Kafka level safety, which uses the page cache. So a DC-wide power loss would be an example.

Basically it would require a correlated failure which is uncommon but does happen. The chance of an uncorrelated failure (two random servers dying at the same time for example) leading to data loss is extremely low.

[fpj]

It sounds right to me that it provides a Kafka-like default semantics. For me, it is an advantage that BK has to be fast while guaranteeing durability. Weakening durability is not desirable from my perspective.

[merlimat]

Durable & fast writes are one of the biggest advantages of BK.

Having said that, the impact of a small amount of data loss varies a lot, depending on the use case and the nature of the data. In some cases, it would make sense to have the option for a less durable mode if that means a reduction in hardware cost.

Another consideration can be made on the usefulness of journal when using locally attached disks in cloud VMs. Since the volume is going to be lost when the VM fails, the journal and the fsync on it will not have the same results they have on a bare metal deployment, where the data is safe unless there's a mechanical failure on the disk.

[jvrao]

With ELPL, I think we can have ledger level durability and still avoid journals.

[Vanlightly]

You probably could but it wouldn't support large numbers of concurrent ledgers very well. The problem is that even with SSDs, you suffer the penalty of many small writes. You can't even delay the syncs to disk as it would increase write latency too much. So while ELPL, once further matured could provide a way for running without the journal, it may not be advisable for all workloads.

[jvrao]

Public clouds charge by the IOPS. This could be another argument we could make. Journal+Index+EntryLog; we have great scope to reduce the storage and IOPS requirements.

[jvrao]

Adding ensemble to the list makes it complete.

[Vanlightly]

I will rename ALL to Ensemble as that is what I meant by that.

[jvrao]

With the current mechanism: An entry will be flushed to the disk before responding. So even it loses unfushed operations it should not lose either E1 or fencing request.
Also at least in salesforce we won't let the bookie comeup if all disks are cleared.

[Vanlightly]

This describes running without a journal, flushing occurs on entry log rotation. But it is true that the cookie mechanism prevents coming back up with wiped disks, so I will remove that.

[jvrao]

You mean the data that is flushed to disk is lost here right? So real data loss.

[Vanlightly]

No, this is describing running without the journal where flushes only occur on entry log rotation. So unflushed data is lost.

[jvrao]

Yes, in case of mismatch, the cookie gets rewritten after all ledgers have been fenced and non-closed

How can you fence a non-closed ledger? Writer might have changed the ensemble and picked some other bookie to continue and may even come back to this bookie after another ensemble change. I may have to read down your exact proposal on how to detect and correct/fix.

running bookie with the data on ephemeral storage

Imagine a situation where Writer is writing to ephemeral storage with Qa=2 Qw=3 En=3 (B0, B1, B2). If a Bookie (B0) goes down and comes up with loss of data (ephemeral) and if that bookie fences can we make the writer continue to write the ledger as it could cotinue to receive 2 acks from other two bookies (B1, B2)?

[jvrao]

Did we consider ELPL to get-rid of journal? With ELPL it is kind of each ledger having its own consistency. With the addition of ledger level flags of durability levels with flush on close option that @dlg99 mentioned we can achieve all the current durability with ledger level granularity and completely get rid of journal.

[jvrao]

If it were a DC power outage

There are many k8s scenarios where the orderly shutdown may not happen. There are various scenarios but I guess we can improve ops process to make it happen. But that would take some effort.

[jvrao]

Based on how exactly you are going to detect unclean shutdown, it may not say if the data loss actually happened or not. Fencing all ledgers may be a big hammer.

[Vanlightly]

We set a bit in the index on start-up and clear it as the last step of shutdown. Based on the value on start-up we know if the bookie was shutdown cleanly. The only way to know if data loss has occurred are the subsequent steps of the process.

[jvrao]

With ELPL, I think we can have ledger level durability and still avoid journals.

[jvrao]

This check runs against the index,

Index file on the disk is also might not have flushed. right? Unless you are talking with ZK metadata comparison.

[eolivelli]

@Vanlightly what's the status of this great work ?

It would be great to move forward with this discussion and also see a preview of the implementation

[Vanlightly]

@eolivelli Once I am done with a couple of active projects I can finish up the code changes and submit a PR for review.

[eolivelli]

I am marking this PR as "approved", as I want to remove "Request changes".
the discussion is broad.

we could also have a one-shot community meeting to discuss about this face-to-face

[Vanlightly]

I have updated the BP number to 46 as I used 44 also for the USE metrics BP.