Quota Integration

[aravindavk]

Integrate Quota feature with Glusterd2

Existing CLI

 $gluster volume quota help

gluster quota commands

volume inode-quota <VOLNAME> enable - Enable/disable inode-quota for <VOLNAME>
volume quota <VOLNAME> {alert-time|soft-timeout|hard-timeout} {<time>} - Set quota timeout for <VOLNAME>
volume quota <VOLNAME> {enable|disable|list [<path> ...]| list-objects [<path> ...] | remove <path>|     remove-objects <path> | default-soft-limit <percent>} - Enable/disable and configure quota for <VOLNAME>
volume quota <VOLNAME> {limit-objects <path> <number> [<percent>]} - Set the maximum number of entries allowed in <path> for <VOLNAME>
volume quota <VOLNAME> {limit-usage <path> <size> [<percent>]} - Set maximum size for <path> for <VOLNAME>
volume quota help - display help for volume quota commands

Need to come up with the list of steps/transactions involved for each command.

[sanoj-unnikrishnan]

we do not need inode-quota enable / disable separately.
quota enable command also enables inode-quota

there is no separate option for inode-quota in quota translator (It exists in marker though).
Currently,

1. enabling quota also enables inode-quota and
2. quota has to be enabled to use inode-quota
   Also, there is no disable option for inode-quota alone. So the command serves no useful purpose.

Testing logs (validated above in source as well):

[root@dhcp35-100 glusterfs]# gluster v inode-quota v1 enable
quota command failed : Quota is disabled, please enable quota

[root@dhcp35-100 glusterfs]# gluster v quota v1 enable
volume quota : success

[root@dhcp35-100 glusterfs]# gluster v inode-quota v1 enable
quota command failed : Inode Quota is already enabled

[root@dhcp35-100 glusterfs]# gluster v inode-quota v1 disable
Invalid quota option : disable

Will reply the behaviour of other commands in subsequent comments

[sanoj-unnikrishnan]

Quota uses a quota.conf file to save gfid of all the directories where the quota limits has been set/configured. This will help quota list command to find all directories with limit set (without doing a crawl ). File is not a very efficient for the purpose. We could explore other methods

cli_quota list_all:
This command lists all the directories that has quota limit set. It reads indvidual gfid from quota.conf and gets the path for the gfid and limit by doing an GF_AGGREGATOR_GETLIMIT RPC to quotad.
cli_cmd_quota_cbk -> cli_cmd_quota_handle_list_all -> read indivaidual gfid from quota.conf and do an rpc to quotad :cli_quotad_getlimit (GF_AGGREGATOR_GETLIMIT ) -> cli_quotad_getlimit_cbk -> print the list

cli_quota_list
If the path is given then in op_stage_quota phase a temporary auxilary mount is created. Subsequently the cli does a getxattr to obtain the limit configured and displays accordingly.
THe aux mount is unmounted after cli does the listing
cli_cmd_quota_cbk -> gf_cli_quota ->glusterd_op_quota -> glusterd_op_stage_quota ->...glusterd_op_quota....-> gf_cli_quota_cbk->gf_cli_quota_list->print_quota_list_from_mountdir->getxattr...

quota limit-usage/ limit-objects :
-> staging phase:
create aux mount
get the gfid for the path from by reading the backend brick xattr
->commit phase:
Fetch existing soft limit percentage by reading xattr
update hard and soft limit by doing setxattr on the auxilary mount
add the gfid entry in the quota.conf if not previously present
remove aux mount

quota remove/remove-objects :
-> staging phase:
create aux mount
get the gfid for the path from by reading the backend brick xattr
->commit phase:
remove the limit xattr on the auxilary mount
remove the gfid entry in the quota.conf if not previously present
remove aux mount

Will cover remaining commands in next comment

[aravindavk]

Thanks for providing more details.

Quota uses a quota.conf file to save gfid of all the directories where the quota limits has been set/configured. This will help quota list command to find all directories with limit set (without doing a crawl ). File is not a very efficient for the purpose. We could explore other methods

Is this file stored in all nodes? If yes, how this will be updated in all nodes when a new entry is added or removed.

cli_quota list_all:

• Does this required aux mount?
• to be run on all Volume nodes or one node?

cli_quota_list
quota limit-usage/ limit-objects :
quota remove/remove-objects :

• If I understand right, this need to be run in any one node.(On the CLI initiated node). Is that correct?
• quota.conf need to be synced to all nodes?

[sanoj-unnikrishnan]

Is this file stored in all nodes? If yes, how this will be updated in all nodes when a new entry is added or removed.

Yes, the files are individually modified on each node by its local glusterd (during commit phase). As discussed we could keep this info with etcd (would resolve couple of issues with keeping quota.conf).

Does this required aux mount?
to be run on all Volume nodes or one node?

"quota list all " does not require aux mount , However "quota list requires one.
The difference is when we do list all, we need all directories with limits, which we get from quota.conf. But to get the to path from gfid in it we need to do a lookup ancestry_path lookup. So for command "quota list all", we get the size and ancestry info by doing a rpc to quotad for each gfid.
For "quota list" comand we have path supplied at command line, so doing an aux mount and fetching size suffices.

If I understand right, this need to be run in any one node.(On the CLI initiated node). Is that correct? quota.conf need to be synced to all nodes?

Correct.

[sanoj-unnikrishnan]

quota enable

• store the quota configuration (quota.conf)
• start quotad (glusterd_quotad_op)
• set the required volume options (server-quota, inode-quota , quota-deem-statfs)
• Increment the quota-xattr version and save it in volinfo
• start crawler on all bricks

quota disable

• clean the quota configuration file (quota.conf)
• stop quotad (glusterd_quotad_op) if it is the only volume with quota enabled. Otherwise trigerr a PROC_START rpc on quotad
• clear the volume options related to quota
• crawler to clear xattr

Note: The crawler can take time proportionate to FS size, so the command must return once crawler is started. One crawler is spawned for each brick in the volume.

Quota contri and size xattrs are versioned. The version is incremented on enable operations. THis is because disable crawl can take long time. (so u can have a disable cleaning older version xattr while the volume is at a newer quota-xattr version).

** alert-time|soft-timeout|hard-timeout **
These are set via volume options after basic input validation

[aravindavk]

Note: The crawler can take time proportionate to FS size, so the command must return once crawler is started. One crawler is spawned for each brick in the volume.

Crawler can be started as daemon, also we can introduce crawler status API since we can get daemon state and crawler can maintain status file.

[aravindavk]

Based on all the inputs from Sanoj, summary of Gd2 and Quota integration is as bellow, Please add if any steps/validations missing

Quota Enable

POST /quota/:volname

Validations:

• Quota already enabled
• Volume exists

Transaction steps

• Step 1: (All Volume nodes) Store the quota configuration (quota.conf) [Question: Can this be stored in etcd?]
• Step 2: (All Volume nodes) Start Quotad
• Step 3: (In the initiated node)
  Set the required volume options (server-quota, inode-quota , quota-deem-statfs)
  Increment the quota-xattr version and save it in volinfo
• Step 4: (All Volume nodes) Start Crawler as daemon on all nodes

Note: Step 1 and 2 can be merged to single transaction step, if step 3 can be executed first then all the other steps can be combined into single Transaction step

Quota Disable

DELETE /quota/:volname

Validations:

• Quota already disabled
• Volume exists

Transaction steps

• Step 1: (All Volume nodes) Clean the quota configuration(quota.conf)
• Step 2: (All Volume nodes) stop quotad (glusterd_quotad_op) if it is the only volume with quota enabled. Otherwise trigerr a PROC_START rpc on quotad
• Step 3: (In the initiated node) clear the volume options related to quota
• Step 4: (All Volume nodes) crawler to clear xattr

Note: Step 1 and 2 can be merged to single transaction step, if step 3 can be executed first then all the other steps can be combined into single Transaction step

Quota List

GET /quota/:volname

Optional parameters to GET request

path=<PATH>

Transaction steps (If path(s) not specified)

Step 1: (Initiated node) Read all GFIDs from quota.conf and RPC to quotad GF_AGGREGATOR_GETLIMIT
Aggregate the result and return

Transaction steps (If path(s) specified)

Step 1: (Initiated node) Create temp aux mount and getxattr on the given path and return

Limit Usage

POST /quota/:volname/paths

Transaction steps

Step 1: (Initiated node)

• Create aux mount and get GFID from given path
• Fetch existing soft limit percentage by reading xattr
• update hard and soft limit by doing setxattr on the auxilary mount
• remove aux mount

Step 2: (All Volume nodes) add the gfid entry in the quota.conf if not previously present

Note: If GFID stored in etcd, then Step 2 can be limited to initiated node only

Limit Objects

POST /quota/:volname/objects

Step 1: (Initiated node)

• Create aux mount and get GFID from given path
• Fetch existing soft limit percentage by reading xattr
• update hard and soft limit by doing setxattr on the auxilary mount
• remove aux mount

Step 2: (All Volume nodes) add the gfid entry in the quota.conf if not previously present

Note: If GFID stored in etcd, then Step 2 can be limited to initiated node only

Remove Quota

DELETE /quota/:volname/paths
DELETE /quota/:volname/objects

Transaction steps

Step 1: (Initiated node)

• Aux mount
• remove the limit xattr on the auxilary mount
• remove aux mount

Step 2: (All volume nodes) remove the gfid entry in the quota.conf if not previously present

Options Set

POST /quota/:volname/options

Transaction steps

Wrapper arount Volume Set

Options Get

GET /quota/:volname/options

Transaction steps

Wrapper arount Volume Get

Options Reset

DELETE /quota/:volname/options

Transaction steps

Wrapper arount Volume Reset

[atinmu]

Checklist

• ReST APIs availibility in wiki
• CLI commands
• Sufficient logging
• Availibility of metrics data if applicable
• Xlator options table upto date?
• Test cases & test coverage of the functionality