Support configurable extra fields for LazyNeMoTarredIterator by pzelasko · Pull Request #9548 · NVIDIA-NeMo/NeMo

pzelasko · 2024-06-26T19:31:26Z

What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

Collection: [Note which collection this PR will affect]

Changelog

Add specific line by line info of high level changes in this PR.

Usage

You can potentially add a usage example below

# Add a code snippet demonstrating how to use this

GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

Before your PR is "Ready for review"

Pre checks:

Make sure you read and followed Contributor guidelines
Did you write any new necessary tests?
Did you add or update any necessary documentation?
Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
- Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

New Feature
Bugfix
Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

Related to # (issue)

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

nemo/lightning/io/mixin.py

+        try:
+            if isinstance(getattr(cfg, attr), fdl.Config):
+                _artifact_transform(getattr(cfg, attr), output_path=output_path)
+        except ValueError:


nemo/collections/llm/tokenizer.py

+
+    track_io(SentencePieceTokenizer, artifacts=[FileArtifact("model_path")])
+    __all__.append("SentencePieceTokenizer")
+except ImportError:


nemo/collections/llm/tokenizer.py

+        ],
+    )
+    __all__.append("AutoTokenizer")
+except ImportError:


nemo/lightning/pytorch/optim/megatron.py

@@ -1,4 +1,4 @@
-from typing import Callable, List, Optional
+from typing import Any, Callable, List, Mapping, Optional


Signed-off-by: Piotr Żelasko <petezor@gmail.com>

krishnacpuvvada · 2024-06-28T22:51:21Z

nemo/collections/common/data/lhotse/nemo_adapters.py

-            seed = resolve_seed(self.shard_seed)
            random.Random(seed).shuffle(shard_ids)

+        # Propagate the random seed


just for reproducibility or is there any other reason?

Reproducibility/consistent RNG behavior across all dataloading modules.

krishnacpuvvada · 2024-06-28T22:58:30Z

nemo/collections/common/data/lhotse/nemo_adapters.py

+
+        >>> cuts = lhotse.CutSet(LazyNeMoIterator(
+        ...     "nemo_manifests/train.json",
+        ...     extra_fields=[{"type": "text_iter", "name": "question", "path": "questions.txt"}],


is there value in allowing to path to be a list of questions also ?

I'm not sure I understood your comment - do you mean "path": ["questions1.txt", "questions2.txt"]? what would be the expected behavior?

github-actions · 2024-07-16T01:51:00Z

This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days.

zhehuaichen

LGTM. Thanks!

* Support configurable extra fields for LazyNeMoTarredIterator Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Add tests and fixes Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Documentation, more tests Signed-off-by: Piotr Żelasko <petezor@gmail.com> --------- Signed-off-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Tugrul Konuk <ertkonuk@gmail.com>

…NeMo#9548) * Support configurable extra fields for LazyNeMoTarredIterator Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Add tests and fixes Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Documentation, more tests Signed-off-by: Piotr Żelasko <petezor@gmail.com> --------- Signed-off-by: Piotr Żelasko <petezor@gmail.com>

* Support configurable extra fields for LazyNeMoTarredIterator Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Add tests and fixes Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Documentation, more tests Signed-off-by: Piotr Żelasko <petezor@gmail.com> --------- Signed-off-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* Support configurable extra fields for LazyNeMoTarredIterator Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Add tests and fixes Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Documentation, more tests Signed-off-by: Piotr Żelasko <petezor@gmail.com> --------- Signed-off-by: Piotr Żelasko <petezor@gmail.com>

…NeMo#9548) * Support configurable extra fields for LazyNeMoTarredIterator Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Add tests and fixes Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Documentation, more tests Signed-off-by: Piotr Żelasko <petezor@gmail.com> --------- Signed-off-by: Piotr Żelasko <petezor@gmail.com> Signed-off-by: Hainan Xu <hainanx@nvidia.com>

…NeMo#9548) * Support configurable extra fields for LazyNeMoTarredIterator Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Add tests and fixes Signed-off-by: Piotr Żelasko <petezor@gmail.com> * Documentation, more tests Signed-off-by: Piotr Żelasko <petezor@gmail.com> --------- Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Support configurable extra fields for LazyNeMoTarredIterator

a5ad998

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

github-actions bot added the common label Jun 26, 2024

pzelasko added 2 commits June 28, 2024 13:28

Add tests and fixes

bdde8f0

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

Merge branch 'main' into extra-field-support-nemo-tar

f20b23e

github-actions bot added core Changes to NeMo Core TTS ASR NLP CI Multi Modal labels Jun 28, 2024

pzelasko marked this pull request as ready for review June 28, 2024 20:18

github-advanced-security bot found potential problems Jun 28, 2024

View reviewed changes

Documentation, more tests

43559ef

Signed-off-by: Piotr Żelasko <petezor@gmail.com>

pzelasko force-pushed the extra-field-support-nemo-tar branch from b00c28c to 43559ef Compare June 28, 2024 20:28

github-actions bot removed core Changes to NeMo Core TTS ASR NLP CI Multi Modal labels Jun 28, 2024

pzelasko added the Run CICD label Jun 28, 2024

pzelasko requested review from krishnacpuvvada and zhehuaichen June 28, 2024 20:29

krishnacpuvvada reviewed Jun 28, 2024

View reviewed changes

github-actions bot added the stale label Jul 16, 2024

zhehuaichen approved these changes Jul 19, 2024

View reviewed changes

pzelasko merged commit e244cfd into main Jul 19, 2024

pzelasko deleted the extra-field-support-nemo-tar branch July 19, 2024 01:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support configurable extra fields for LazyNeMoTarredIterator#9548

Support configurable extra fields for LazyNeMoTarredIterator#9548
pzelasko merged 4 commits intomainfrom
extra-field-support-nemo-tar

pzelasko commented Jun 26, 2024

Uh oh!

Check notice

Check notice

Check notice

Check notice

krishnacpuvvada Jun 28, 2024

Uh oh!

pzelasko Jul 1, 2024

Uh oh!

krishnacpuvvada Jun 28, 2024

Uh oh!

pzelasko Jul 1, 2024

Uh oh!

github-actions bot commented Jul 16, 2024

Uh oh!

zhehuaichen left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

		@@ -1,4 +1,4 @@
		from typing import Callable, List, Optional
		from typing import Any, Callable, List, Mapping, Optional

Conversation

pzelasko commented Jun 26, 2024

What does this PR do ?

Changelog

Usage

GitHub Actions CI

Before your PR is "Ready for review"

Who can review?

Additional Information

Uh oh!

Check notice

Check notice

Check notice

Check notice

krishnacpuvvada Jun 28, 2024

Choose a reason for hiding this comment

Uh oh!

pzelasko Jul 1, 2024

Choose a reason for hiding this comment

Uh oh!

krishnacpuvvada Jun 28, 2024

Choose a reason for hiding this comment

Uh oh!

pzelasko Jul 1, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jul 16, 2024

Uh oh!

zhehuaichen left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments