[QC-670] Add parameter to limit the size of the objects sent to the ccdb #8013
Conversation
Please consider the following formatting changes to AliceO2Group#8013
|
Shouldn't this be enforced on the server side? @costing |
|
I think it would be more efficient to not send it at all than to be rejected after the transfer. But yes, the server could also impose them. Just that a client could have a much finer grained policy on what is acceptable or not in each situation. The question on how to signal this to the client applies in both cases. What if instead of an exception the return values of these methods would change from |
|
@ktf @costing A limit on the server-side is a last resort safety. I think that it should also be implemented. Good idea @ktf. Nevertheless, we need a finer grain system because different tasks have different needs. In the QC, we also consider having different size for different data types. Concerning the return value, I am ok with it. It is more acceptable than an exception in the sense that the current clients of the code would not be affected. @ktf what do you think ? |
|
I am not sure I get the why doing it client side is better. It means that the limit will be hardcoded in the C++ and therefore old releases will continue to work with a badly set / not set limit. Adjusting fine grained limits will mean a new release. If that's what we need, a fine grained rejection list on the server side seems to be the most flexible solution. |
|
That said, I personally do not like exceptions in core classes, but I have no strong preference for this API, if really needed. |
|
For the record: some calibration workflows may need to upload huge objects to in-memory transient CCDB (for the transfer to the DCS) |
|
@ktf the value(s) will be taken from the config files not hard-coded. |
Please consider the following formatting changes to AliceO2Group#8013
|
I have added the return value and a test. |
|
what is going on with the mac builder ? |
|
@Barthelemy @ktf indeed the server can reject requests above a set threshold. It is now (tag 1.0.15, just done) configurable via the For finer grained policy (i.e. per path) the request would have to be first parsed, so the transfer is going to happen anyway. Thus I agree with Barth that this should only be used as a safety net only. |
|
You would just need to parse the headers, no? You would need to do that in any case, right? A big object is hopefully the exception not the rule. HTTP does not require you to get the payload before you return an error code, IIRC. |
|
That said, if @Barthelemy is happy, I am happy. |
|
@ktf in principle yes, in practice I don't implement the full server but rely on Tomcat to do the heavy lifting. The callback to the CCDB code is after the request is handled by Tomcat. |
|
Why is macos failing ? can we merge ? |
|
Yes, @MichaelLettrich @TimoWilken are already on the macOS issue (since last week actually). |
5 participants
We have encountered cases when the users would send very large objects to the CCDB and causing havoc later on. We would like to limit the size of the objects sent to the QCDB.
I propose to add a parameter to the
store*methods specifying such a maximum limit.Doing it on the client side, i.e. in the QC, would mean that we have to do the gymnastic of serializing the object just to know its size although it is done any ways in the CcdbApi. Therfore, I prefer to do it in the ccdbapi to avoid doing it twice in particular given that it is a very expensive operation.
I also considered to have the max size as an object member. However one might want to vary the max size of the objects.
Last note: the logging of the message when an object is too big might be an issue as it could be repeated many times. On the other hand it is a debug message and it is quite important to be informed that the operation failed. I considered throwing an exception but it would change the interface of these methods and I was not sure it would be wise.
What do you think ?