Skip to content

Update instructions for AWS credentials in ArXiv download and extract tutorial#1380

Merged
sarahyurick merged 6 commits intoNVIDIA-NeMo:mainfrom
sarahyurick:arxiv_aws_credentials
Jan 20, 2026
Merged

Update instructions for AWS credentials in ArXiv download and extract tutorial#1380
sarahyurick merged 6 commits intoNVIDIA-NeMo:mainfrom
sarahyurick:arxiv_aws_credentials

Conversation

@sarahyurick
Copy link
Copy Markdown
Contributor

No description provided.

… tutorial

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Jan 15, 2026

Greptile Summary

This PR improves the ArXiv tutorial by moving AWS credentials setup to the top of the notebook, before Ray client initialization. This prevents credentials-related errors that can occur when credentials are set after Ray starts. The changes include:

  • Added AWS credentials cell at the beginning of the notebook with clear instructions
  • Added helpful note about not using quotes when setting environment variables
  • Moved credentials instructions from the ArXiv section to the Prerequisites section
  • Replaced shell command (!s5cmd) with subprocess.run() for better error handling and output capture in the S3 bucket accessibility test
  • Added # noqa: S602 comment to suppress security warning for necessary shell=True usage

Confidence Score: 5/5

  • This PR is safe to merge with minimal risk
  • The changes are documentation and tutorial improvements that fix a real user experience issue. The code changes are minimal and well-justified (replacing shell command with subprocess for better error handling). No logical errors or security issues were introduced.
  • No files require special attention

Important Files Changed

Filename Overview
tutorials/text/download-and-extract/download_extract_tutorial.ipynb Moved AWS credentials setup to top of notebook before Ray initialization, added helpful instructions about credential formatting, and improved S3 bucket accessibility test with better error handling

Sequence Diagram

sequenceDiagram
    participant User
    participant Notebook
    participant AWS
    participant Ray
    participant ArxivPipeline

    User->>Notebook: Set AWS_ACCESS_KEY_ID
    User->>Notebook: Set AWS_SECRET_ACCESS_KEY
    User->>Notebook: Set AWS_SESSION_TOKEN
    Note over Notebook: Credentials must be set BEFORE Ray client
    
    Notebook->>Ray: Initialize RayClient()
    Ray-->>Notebook: Client ready
    
    Notebook->>AWS: Test S3 bucket access (subprocess.run)
    AWS-->>Notebook: Return bucket listing
    
    Notebook->>ArxivPipeline: Create ArxivDownloadExtractStage
    Notebook->>ArxivPipeline: Run pipeline
    ArxivPipeline->>AWS: Download tar files (uses credentials)
    AWS-->>ArxivPipeline: Return data
    ArxivPipeline-->>Notebook: Return extracted results
Loading

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
@sarahyurick sarahyurick added the r1.1.0 Pick this label for auto cherry-picking into r1.1.0 label Jan 15, 2026
@sarahyurick sarahyurick merged commit 058fa93 into NVIDIA-NeMo:main Jan 20, 2026
18 checks passed
pablo-garay pushed a commit that referenced this pull request Jan 20, 2026
… tutorial (#1380)

* Update instructions for AWS credentials in ArXiv download and extract tutorial

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

* ruff

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

---------

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
sarahyurick added a commit that referenced this pull request Jan 20, 2026
… tutorial (#1380) (#1402)

* Update instructions for AWS credentials in ArXiv download and extract tutorial



* ruff



---------

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>
@sarahyurick sarahyurick deleted the arxiv_aws_credentials branch February 9, 2026 18:13
lbliii pushed a commit to lbliii/NeMo-Curator that referenced this pull request Mar 16, 2026
… tutorial (NVIDIA-NeMo#1380) (NVIDIA-NeMo#1402)

* Update instructions for AWS credentials in ArXiv download and extract tutorial

* ruff

---------

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
lbliii added a commit that referenced this pull request Mar 23, 2026
* ci: Bump version to 1.1.0 (#1364) (#1365)

Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Dong Hyuk Chang <thomaschang26@tutanota.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* feat: FFmpeg to 8.0.1 (#1362) (#1363)

Signed-off-by: Ao Tang <aot@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Ao Tang <aot@nvidia.com>
Co-authored-by: Ayush Dattagupta <ayushdg95@gmail.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* Fix bug in SDG example (#1370) (#1371)

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* Fix bug in Gliner tutorial (#1372) (#1378)

* Fix bug in Gliner tutorial

* update readmes

---------

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* Address aiohttp and urllib3 cve (#1379) (#1383)

Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Dong Hyuk Chang <thomaschang26@tutanota.com>
Co-authored-by: Ayush Dattagupta <ayushdg95@gmail.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* Adding one worker per partition to FilePartioningStage and URLGeneratorStage (#1350) (#1366)

Signed-off-by: Abhinav Garg <abhinavg@stanford.edu>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Abhinav Garg <abhinavg@stanford.edu>
Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>
Co-authored-by: Ayush Dattagupta <ayushdg95@gmail.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* Update instructions for AWS credentials in ArXiv download and extract tutorial (#1380) (#1402)

* Update instructions for AWS credentials in ArXiv download and extract tutorial

* ruff

---------

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* cp: Revert "Remove nvenc/dec for xenna 0.1.6 (#1202)" (#1374) (#1403)

This reverts commit c4805ae.

Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* Pin sklearn to < 1.8.0 for cuml 25.10 for r.1.1.0 #1405

Signed-off-by: Lawrence Lane <llane@nvidia.com>

* Clarify instructions for downloading the Llama Nemotron Post-Training Dataset (#1416) (#1423)

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* CP: Fix vllm API compatibility with Video Pipeline + Upgrade vLLM to 0.14 (#1429)

* vllm API compatibility fixed

Signed-off-by: Ao Tang <aot@nvidia.com>

* upgrade vllm to 0.14.0

Signed-off-by: Ao Tang <aot@nvidia.com>

* refactor

Signed-off-by: Ao Tang <aot@nvidia.com>

* pyproject update

Signed-off-by: Ao Tang <aot@nvidia.com>

* add protobuf in constraint-dependencies

Signed-off-by: Ao Tang <aot@nvidia.com>

* comment improve

Signed-off-by: Ao Tang <aot@nvidia.com>

* resolve pyproject

Signed-off-by: Ao Tang <aot@nvidia.com>

---------

Signed-off-by: Ao Tang <aot@nvidia.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* ci: Address setuptools CVE (#1438) (#1439)

* Address CVE fixes

* Remove cache of aiohttp from ray

* Update uv lock

* Update cache path

---------

Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* ci: Optimize docker layer and uv with no cache (#1444) (#1446)

* Optimize docker layer and uv with no cache

* Add missing slash

* Add comments to dockerfile

---------

Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* Purge InternVideo2 (#1451) (#1462)

* Remove Internvideo2

* more to remove

* fix writer

* Enhance Clip class to include cosmos_embed1_frames and cosmos_embed1_embedding in total size calculation

* remove iv2

---------

Signed-off-by: Ao Tang <aot@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Ao Tang <aot@nvidia.com>
Co-authored-by: Ayush Dattagupta <ayushdg95@gmail.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* ci: Update cve for python-multipart (#1450) (#1455)

* Update cve for python-multipart

* Update uv lock

---------

Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com>
Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* cherry pick commit, no benchmarking needed (#1461)

Co-authored-by: Huy Vu2 <huvu@login-eos02.eos.clusters.nvidia.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* ci: Update vllm to 0.14.1 and override conflict (#1467) (#1468)

* Update vllm to 0.14.1 and override conflict

* Upperbound numpy for Numba compatibility

* Update vllm to 0.15.1

---------

Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* ci: Remove thirdparty aiohttp file from ray (#1469) (#1475)

Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* Fix: fasttext predict call for numpy>2 (#1482) (#1486)

Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Ayush Dattagupta <ayushdg95@gmail.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* Update transformers dependency to exact version 4.55.2 in pyproject.t… (#1471) (#1488)

* Update transformers dependency to exact version 4.55.2 in pyproject.toml and uv.lock to prevent import failures in Cosmos Embed. Downgrade tokenizers version to 0.21.4 for compatibility.

* Update transformers dependency in pyproject.toml and uv.lock to allow versions up to 4.55.2, ensuring compatibility with Cosmos Embed imports.

---------

Signed-off-by: Abhinav Garg <abhinavg@stanford.edu>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Abhinav Garg <abhinavg@stanford.edu>
Co-authored-by: Ayush Dattagupta <ayushdg95@gmail.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* Cherry pick `tutorials` changes from #1477 (#1491)

* Update tutorial README

Signed-off-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>

* Update quickstart

Updated sample sentences to provide more detailed feedback.

Signed-off-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>

---------

Signed-off-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* Update tutorial to be more explicit about num_gpus (#1492) (#1499)

* Update tutorial to be more explicit about num_gpus

* fix false positive secret scan

* Hopefully fix the secrets

---------

Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com>
Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com>
Co-authored-by: Ayush Dattagupta <ayushdg95@gmail.com>
Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* Add relevant 26.02 docs to r1.1.0 (#1493)

* add release notes

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

* add more pages

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

* add more pages

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

* add new sdg docs

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

* update remaining files from sdg docs

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

* continue adding more changes

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

* more video docs

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

* add remaining updates

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>

---------

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* ci: Address new CVEs from rc4 (#1497) (#1500)

* Scrub thirdparty aiohttp file from ray

* Address new rc4 CVE

* Apt get for consistency

---------

Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Ayush Dattagupta <ayushdg95@gmail.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* Add feedback to tutorials (#1476) (#1501)

* Add feedback to tutorials

* clarify install instructions for classifier tutorials

* byo classifiers

* add descriptions

---------

Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
Signed-off-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* ci: Update pyasn1 in uv lock (#1505)

Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* Refactor video frame extraction to improve PyNvCodec availability check (#1511) (#1513)

* Refactor video frame extraction to improve PyNvCodec availability check

- Removed the try-except block for importing PyNvcFrameExtractor, simplifying the import logic.
- Updated the condition for initializing the PyNvcFrameExtractor in the VideoFrameExtractionStage to rely solely on the _PYNVC_AVAILABLE flag.
- Adjusted the handling of pixel format conversion in NvVideoDecoder to prepare for future updates to cvcuda.

* Refactor NvVideoDecoder to replace deprecated nvcv_image with cvcuda tensor

- Updated NvVideoDecoder to remove the use of nvcv_image, which is deprecated, and replaced it with cvcuda tensor.
- Adjusted related tensor operations and tests to ensure compatibility with the new cvcuda implementation.

* Update import statements in test_nvcodec_utils.py to include ruff linting rule

- Modified import statements in the test file to include the RUF100 linting rule, ensuring better adherence to coding standards.
- This change enhances the clarity of the import handling tests.

* Update tests/utils/test_nvcodec_utils.py

* Update tests/utils/test_nvcodec_utils.py

---------

Signed-off-by: Abhinav Garg <abhinavg@stanford.edu>
Signed-off-by: [Your Name] <your.email@example.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Co-authored-by: Abhinav Garg <abhinavg@stanford.edu>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Ayush Dattagupta <ayushdg95@gmail.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* docs: isolate release notes and changelog (#1529)

* docs: isolate release notes and changelog

Signed-off-by: Lawrence Lane <llane@nvidia.com>

* abhinav's feedback

Signed-off-by: Lawrence Lane <llane@nvidia.com>

* feedback

Signed-off-by: Lawrence Lane <llane@nvidia.com>

---------

Signed-off-by: Lawrence Lane <llane@nvidia.com>

* ci: Update final release version (#1540)

Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>

* docs: release note updates

Signed-off-by: Lawrence Lane <llane@nvidia.com>

---------

Signed-off-by: Dong Hyuk Chang <donghyukc@nvidia.com>
Signed-off-by: NeMo Bot <nemo-bot@nvidia.com>
Signed-off-by: Lawrence Lane <llane@nvidia.com>
Signed-off-by: Ao Tang <aot@nvidia.com>
Signed-off-by: Sarah Yurick <sarahyurick@gmail.com>
Signed-off-by: Abhinav Garg <abhinavg@stanford.edu>
Signed-off-by: Ayush Dattagupta <ayushdg95@gmail.com>
Signed-off-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>
Signed-off-by: [Your Name] <your.email@example.com>
Co-authored-by: Pablo Garay <palenq@gmail.com>
Co-authored-by: Dong Hyuk Chang <thomaschang26@tutanota.com>
Co-authored-by: Ao Tang <aot@nvidia.com>
Co-authored-by: Ayush Dattagupta <ayushdg95@gmail.com>
Co-authored-by: Sarah Yurick <53962159+sarahyurick@users.noreply.github.com>
Co-authored-by: Abhinav Garg <abhinavg@stanford.edu>
Co-authored-by: Praateek Mahajan <praateekmahajan@users.noreply.github.com>
Co-authored-by: Huy Vu <86480512+huvunvidia@users.noreply.github.com>
Co-authored-by: Huy Vu2 <huvu@login-eos02.eos.clusters.nvidia.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

r1.1.0 Pick this label for auto cherry-picking into r1.1.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants