feat: introduce per-sync configurations

[james-milligan]

This PR

• introduces the SyncProviderConfig object and associated start flags to pass configuration to specific sync providers

Related Issues

#405

Notes

Follow-up Tasks

How to test

[codecov]

Codecov Report

Merging #448 (2dd6912) into main (6a039ce) will decrease coverage by 6.40%.
The diff coverage is 46.75%.

@@            Coverage Diff             @@
##             main     open-feature/flagd#448      +/-   ##
==========================================
- Coverage   68.47%   62.08%   -6.40%     
==========================================
  Files          13       15       +2     
  Lines        1656     1891     +235     
==========================================
+ Hits         1134     1174      +40     
- Misses        461      654     +193     
- Partials       61       63       +2     

Impacted Files	Coverage Δ
pkg/runtime/runtime.go	0.00% <ø> (ø)
pkg/sync/file/filepath_sync.go	54.54% <ø> (ø)
pkg/sync/grpc/grpc_sync.go	62.04% <ø> (ø)
pkg/sync/http/http_sync.go	43.29% <0.00%> (ø)
pkg/sync/kubernetes/kubernetes_sync.go	83.55% <ø> (-0.08%)	⬇️
pkg/runtime/from_config.go	25.30% <47.36%> (ø)

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[beeme1mr]

I wasn't able to get the examples working. I built flagd using your branch and confirmed the config example flags were in the expected location.

./flagd start --sync-providers=\[{\"uri\":\"config/samples/example_flags.json\"\,\"provider\":\"file\"}\]

                 ______   __       ________   _______    ______      
                /_____/\ /_/\     /_______/\ /______/\  /_____/\     
                \::::_\/_\:\ \    \::: _  \ \\::::__\/__\:::_ \ \    
                 \:\/___/\\:\ \    \::(_)  \ \\:\ /____/\\:\ \ \ \   
                  \:::._\/ \:\ \____\:: __  \ \\:\\_  _\/ \:\ \ \ \  
                   \:\ \    \:\/___/\\:.\ \  \ \\:\_\ \ \  \:\/.:| | 
                    \_\/     \_____\/ \__\/\__\/ \_____\/   \____/_/ 

2023-03-01T14:01:42.843-0500    info    cmd/start.go:104        flagd version: dev (5ecdb19), built at: 2023-03-01T14:00:02Z    {"component": "start"}
2023-03-01T14:01:42.843-0500    fatal   cmd/start.go:144        no sync implementation set      {"component": "start"}
github.com/open-feature/flagd/cmd.glob..func1
        /home/beemer/code/openfeature/flagd/cmd/start.go:144
github.com/spf13/cobra.(*Command).execute
        /home/beemer/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:920
github.com/spf13/cobra.(*Command).ExecuteC
        /home/beemer/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044
github.com/spf13/cobra.(*Command).Execute
        /home/beemer/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968
github.com/open-feature/flagd/cmd.Execute
        /home/beemer/code/openfeature/flagd/cmd/root.go:37
main.main
        /home/beemer/code/openfeature/flagd/main.go:30
runtime.main
        /home/beemer/sdk/go1.19.6/src/runtime/proc.go:250

[Kavindu-Dodan]

Overal looks good 👍

Please check linting failures

[beeme1mr]

It's probably unrelated to this change but I'm seeing two WRITE events in the logs every time a flag configuration changes using the file source. Could you please confirm that this change isn't the root cause. If it's not, please create a follow up issue to investigate.

[james-milligan]

It's probably unrelated to this change but I'm seeing two WRITE events in the logs every time a flag configuration changes using the file source. Could you please confirm that this change isn't the root cause. If it's not, please create a follow up issue to investigate.

Please can you provide more info - im not able to replicate this behaviour:

2023-03-08T09:27:41.069Z        debug   file/filepath_sync.go:121       Configuration config/samples/example_flags.json:  ALL   {"component": "sync", "sync": "filepath"}
2023-03-08T09:28:31.740Z        info    file/filepath_sync.go:78        filepath event: config/samples/example_flags.json CHMOD {"component": "sync", "sync": "filepath"}
2023-03-08T09:28:31.741Z        info    file/filepath_sync.go:78        filepath event: config/samples/example_flags.json WRITE {"component": "sync", "sync": "filepath"}
2023-03-08T09:28:31.741Z        debug   file/filepath_sync.go:121       Configuration config/samples/example_flags.json:  ALL   {"component": "sync", "sync": "filepath"}
2023-03-08T09:28:36.256Z        info    file/filepath_sync.go:78        filepath event: config/samples/example_flags.json CHMOD {"component": "sync", "sync": "filepath"}
2023-03-08T09:28:36.256Z        info    file/filepath_sync.go:78        filepath event: config/samples/example_flags.json WRITE {"component": "sync", "sync": "filepath"}
2023-03-08T09:28:36.256Z        debug   file/filepath_sync.go:121       Configuration config/samples/example_flags.json:  ALL   {"component": "sync", "sync": "filepath"}

I get repeated patterns of CHMOD, WRITE then a configuration ALL event using this startup command:

go run main.go start --sources='[{"uri":"config/samples/example_flags.json","provider":"file"}]' --debug 

[toddbaert]

It's probably unrelated to this change but I'm seeing two WRITE events in the logs every time a flag configuration changes using the file source. Could you please confirm that this change isn't the root cause. If it's not, please create a follow up issue to investigate.

Please can you provide more info - im not able to replicate this behaviour:

2023-03-08T09:27:41.069Z        debug   file/filepath_sync.go:121       Configuration config/samples/example_flags.json:  ALL   {"component": "sync", "sync": "filepath"}
2023-03-08T09:28:31.740Z        info    file/filepath_sync.go:78        filepath event: config/samples/example_flags.json CHMOD {"component": "sync", "sync": "filepath"}
2023-03-08T09:28:31.741Z        info    file/filepath_sync.go:78        filepath event: config/samples/example_flags.json WRITE {"component": "sync", "sync": "filepath"}
2023-03-08T09:28:31.741Z        debug   file/filepath_sync.go:121       Configuration config/samples/example_flags.json:  ALL   {"component": "sync", "sync": "filepath"}
2023-03-08T09:28:36.256Z        info    file/filepath_sync.go:78        filepath event: config/samples/example_flags.json CHMOD {"component": "sync", "sync": "filepath"}
2023-03-08T09:28:36.256Z        info    file/filepath_sync.go:78        filepath event: config/samples/example_flags.json WRITE {"component": "sync", "sync": "filepath"}
2023-03-08T09:28:36.256Z        debug   file/filepath_sync.go:121       Configuration config/samples/example_flags.json:  ALL   {"component": "sync", "sync": "filepath"}

I get repeated patterns of CHMOD, WRITE then a configuration ALL event using this startup command:

go run main.go start --sources='[{"uri":"config/samples/example_flags.json","provider":"file"}]' --debug 

I think this is all old behavior, and not much can be done about it. It seems to be all about how particular editors and programs update files. Even in the most basic cases, programs actually remove/write the file. The only exception is if the file being watched is a symlink and that symlink is updated. That's the only thing that seems to be completely atomic.

Even if I run something like: sed -i 's/"defaultVariant": "on"/"defaultVariant": "off"/g' config/samples/example_flags.json which uses sed's "in-place" option (-i), the file is actually removed and replaced with new contents, resulting in 3 events: CHMOD, REMOVE, WRITE (I had no idea sed worked that way, but apparently it does). I've observed this running on Linux, natively (no docker or WSL involved).

[toddbaert]

Re-tested and re-approved.