Skip to content

Improve ConfigNode load/save perf#88

Merged
NathanKell merged 8 commits into
devfrom
ConfigNodePerf
Aug 26, 2022
Merged

Improve ConfigNode load/save perf#88
NathanKell merged 8 commits into
devfrom
ConfigNodePerf

Conversation

@NathanKell
Copy link
Copy Markdown
Contributor

@NathanKell NathanKell commented Aug 25, 2022

Rewrite PreFormatConfig to iterate through strings less and to support skipping checking for {} and //
Also patch writing for a little bit of extra perf.

The preformat patch includes, disabled, siimav's parallel implementaiton; it's disabled because at the moment it's strictly slower than stock code AFAICT.

This yields up to an 8x boost (or so) in read/write performance for sfs, and lower but still substantial gains elsewhere.

For testing, add this to a file in gamedata and then load craft/saves.

@KSP_COMMUNITY_FIXES:FINAL
{
	@CONFIGNODE_PERF_SKIP_PROCESSING_KEYS
	{
		item = attPos0
	}
	@CONFIGNODE_PERF_SKIP_PROCESSING_SUBSTRINGS
	{
		item = attRot
	}
}

Note that the skip nodes will only apply after MM loads; before that you'll just get the regular speedup.

@NathanKell NathanKell changed the title Rewrite PreFormatConfig to iterate through strings less Improve ConfigNode load/save perf Aug 25, 2022
@NathanKell NathanKell merged commit fb90bee into dev Aug 26, 2022
@NathanKell NathanKell deleted the ConfigNodePerf branch August 26, 2022 04:02
gotmachine pushed a commit that referenced this pull request Dec 7, 2022
* Rewrite PreFormatConfig to iterate through strings as little as possible.

* Improve logging, add @siimav's parallel processing bit (seems to be strictly slower even for 1m+ rows, tho)

* Patch writing, too, for a slight improvement in perf

* Fix a lingering issue with trailing braces

* Patch on startup, not after MM. Skipping keys will only happen after MM, but they're really just there for sfs loading, can't trust what's in gamedatabase enough to skip parsing there.

* Update readme and Settings

* Big improvements to save perf by not writing lines / flushing / using a puny buffer.

* missed a newline
gotmachine pushed a commit that referenced this pull request Jan 1, 2023
* Rewrite PreFormatConfig to iterate through strings as little as possible.

* Improve logging, add @siimav's parallel processing bit (seems to be strictly slower even for 1m+ rows, tho)

* Patch writing, too, for a slight improvement in perf

* Fix a lingering issue with trailing braces

* Patch on startup, not after MM. Skipping keys will only happen after MM, but they're really just there for sfs loading, can't trust what's in gamedatabase enough to skip parsing there.

* Update readme and Settings

* Big improvements to save perf by not writing lines / flushing / using a puny buffer.

* missed a newline
gotmachine pushed a commit that referenced this pull request Jan 30, 2023
* Rewrite PreFormatConfig to iterate through strings as little as possible.

* Improve logging, add @siimav's parallel processing bit (seems to be strictly slower even for 1m+ rows, tho)

* Patch writing, too, for a slight improvement in perf

* Fix a lingering issue with trailing braces

* Patch on startup, not after MM. Skipping keys will only happen after MM, but they're really just there for sfs loading, can't trust what's in gamedatabase enough to skip parsing there.

* Update readme and Settings

* Big improvements to save perf by not writing lines / flushing / using a puny buffer.

* missed a newline
gotmachine pushed a commit that referenced this pull request Jan 30, 2023
* Rewrite PreFormatConfig to iterate through strings as little as possible.

* Improve logging, add @siimav's parallel processing bit (seems to be strictly slower even for 1m+ rows, tho)

* Patch writing, too, for a slight improvement in perf

* Fix a lingering issue with trailing braces

* Patch on startup, not after MM. Skipping keys will only happen after MM, but they're really just there for sfs loading, can't trust what's in gamedatabase enough to skip parsing there.

* Update readme and Settings

* Big improvements to save perf by not writing lines / flushing / using a puny buffer.

* missed a newline
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant