[bookie] provide option to skip adding entry to journal

[rdhabalia]

Motivation

As described at : apache/pulsar#5128

Right now, bookie serves WAL and provides durability by writing entry to journal before acking the message and it persist the entry to entry log file. bookie also provides below options to improve throughput

• to persist entry with fsync disable
• writing entry to separate devices for journal and ledger.

So, persisting entry to journal before acking gives durability but sometimes it becomes bottleneck (mostly due to number of io ops) when user wants to achieve higher throughput in system.

Few users have usecase to achieve high throughput with low publish latency, can compromise with durability and mostly reads from the cache. In that case, if we skip adding entry to journal and acks the message as soon as it's added to ledger cache then user can achieve comparatively very high writes in bookie. and it also helps in pulsar to reduce footprints of bookie while serving such usecases.

So, we want to add option to skip adding entry to journal. We also want to make sure that bookie should be able to handle change in this configuration and should come back gracefully whenever this configuration has been changed.

Changes

Provide option journalSkipEntryEnable using which bookie can skip adding entry to journal while serving addEntry request.

[eolivelli]

I think we should implement this change somehow but the client must be aware that entries are not persisted, otherwise we are totally losing the guarantees of the Bookkeeper.

This is what I would expect:

• a new WriteFlag on the client BYPASS_JOURNAL
• the client won't update the LastAddConfirmed pointer because there is mo guarantee
• Bookie won't write to the journal (fencing??)
• have a new implementation of the force() API that waits (or forces) a flush of the MemTable for the writing ledger (not so easy) and gets a guarantee of durability, so it can advance the LAC
• no ensemble change allowed

This will be mostly like DEFERRED_SYNC WriteFlag but:

• without writes to journal
• force() need a flush of the memtable, not a fsync() on journal
  The LAC part is already implemented.

I think we should discuss this stuff on the mailing list.

There was a BP (Bookkeeper Proposal) about this new feature.

Cc @jvrao @ivankelly @fpj @sijie @reddycharan

[sijie]

Agreed to what @eolivelli commented. In order to implement this feature properly, we have to follow what we have discussed in the BP before. Since this change touches the semantic and guarantees that BookKeeper is provided, a discussion and a BP are required to get the community onboard with this change.

[rdhabalia]

Thanks @eolivelli for your inputs. Actually, we have a flexibility to create a small bookie cluster where we can compromise persistent durability so, I have provided option to introduce flag at bookie-server level.

But I think your proposal is better because in that case client can decide durability level and bookie can serve both durable and non-durable usecase at the same time.

I have also gone through with BP-14 but our usecase is more related to skipping add-entry to journal to achieve higher throughput. So, we will need implementation with flagBYPASS_JOURNAL as you fully described in your previous reply. So, if you have not worked on it then do you think I can create a PR with those changes so, we can use this client-side feature instead compromising durability at bookie-server side.?

[rdhabalia]

@eolivelli I think BP-14 is definitely a good proposal but I think we don't want to mix requests coming from both the usecases to one bookie and want bookie to serve one kind of usecase mainly multiple specifications of usecases to achieve highest throughput.
So, I think adding this server-side option would be worth to have and we can always add BP-14 as an enhancement. any thoughts?

• @merlimat

[sijie]

@rdhabalia

I don't think the main point of BP-14 is to have client flag and allow mixing workloads. The main point of BP-14 to adjust our protocol to make sure it stick to LAC protocol even bookies are running in a less-durable mode. The change you are proposing is different from adding a flag in journal to bypass fsync. A process crash can easily cause data loss which can mess up LAC protocol, which is much much weaker than skipping fsync on journal.

At the minimal, we should do the followings:

• not advance LAC for non-durable writes
• add flush method to flush writes and advance LAC only after flush is called.

[rdhabalia]

At the minimal, we should do the followings: not advance LAC for non-durable writes, add flush method to flush writes and advance LAC only after flush is called.

@sijie
Ok.. so, if we can make LAC related changes in this PR to make sure bookie doesn't mess up with LAC protocol then we should be fine with the PR. is that correct?

[sijie]

so, if we can make LAC related changes in this PR to make sure bookie doesn't mess up with LAC protocol then we should be fine with the PR. is that correct?

If you are not taking the approach proposed in BP-14, it would be good to write a BP for your approach before actually implementing the pull request. Because it is a piece that the community has discussed before and people has something already in their mind. Writing a BP for your approach would help align everyone on this.

[eolivelli]

We must do the implementation on the bookie and then deliver it to the public client API

[eolivelli]

@rdhabalia this is the BP #1944 but it is becoming stale.

[eolivelli]

@rdhabalia what's the status of this patch?

cc @atris

[rdhabalia]

@eolivelli I thought #1944 will take care it so, I was waiting for that PR to be merged. We can close this PR and I can create another new one if #1944 will not address it.

[eolivelli]

this change relates to #2401

[danielnaumau]

I've been searching for a while and I cannot find a proper way how to disable journal in pulsar. What flag should I use now to disable it?