update datasets used in lecture cross_section

[shlff]

Hi @jstac and @mmcky this PR updates datasets used in the lecture cross_section in the lecture python intro series:

• Add a discussion of cross-sectional distributions in economics lecture-python-intro#119

To do:

• add a README.md file to illustrate the datasets

[jstac]

Thanks @shlff. Please ping and work with @mmcky when this is ready for review.

[shlff]

Sure thanks @jstac .

Hi @mmcky this PR is ready for review. Looking forward to your comments.

[mmcky]

@shlff I would suggest you add a table to the README that includes information about the datasets such as

name	description	approx. size
/us_cities.txt	A dataset describing about cities in the US ()	32.4Mb

and then include some instructions on how to fetch the data for use in a lecture (or elsewhere).

These will be stored using git-lfs so can you fetch git-lfs directly in a lecture? Is that the aim. Perhaps we should have a quick zoom about what your requirements are here.

[shlff]

Thanks for your helpful comments @matt ***@***.***>. I will update the table and the instructions. I can confirm that we can fetch git-lfs directly in a lecture. Please find the preview of this PR here: * QuantEcon/lecture-python-intro#123 I'd love for a quick zoom chat if you'd like. Best, Shu *

…

________________________________ From: mmcky ***@***.***> Sent: Monday, April 17, 2023 3:09 PM To: QuantEcon/high_dim_data ***@***.***> Cc: Shu Hu ***@***.***>; Mention ***@***.***> Subject: Re: [QuantEcon/high_dim_data] update datasets used in lecture cross_section (PR #3) @shlff<https://github.com/shlff> I would suggest you add a table to the README that includes information about the datasets such as name description approx. size /us_cities.txt A dataset describing about cities in the US () 32.4Mb and then include some instructions on how to fetch the data for use in a lecture (or elsewhere). These will be stored using git-lfs so can you fetch git-lfs directly in a lecture? Is that the aim. Perhaps we should have a quick zoom about what your requirements are here. — Reply to this email directly, view it on GitHub<#3 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AM3OYIIESWXXI6JCJ6DM23DXBTGBXANCNFSM6AAAAAAW74VD2Q>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[mmcky]

@shlff I see. I'm surprised this works given git-lfs are hashed commits but GitHub must resolve that link to its stored location. That's neat. I wonder what the size limit is.

df_fs = pd.read_csv('https://media.githubusercontent.com/media/QuantEcon/high_dim_data/update_csdata/cross_section/forbes-global2000.csv')

Seems that it is managed by QuantEcon data allowance and bandwidth usage.

https://docs.github.com/en/repositories/working-with-files/managing-large-files/about-storage-and-bandwidth-usage

We have 50Gb of storage and bandwidth so not binding in the medium term. But it will be interesting to see the usage as people start downloading datasets from our GitHub account using python

[shlff]

Thanks for your advice and chat @mmcky . I've created an issue with the bandwidth usage:

• monitoring bandwidth usage #4

I will keep you posted once I update the

• data description,
• instructions,
• review and modify

[mmcky]

thanks @shlff if you're happy with this PR please go ahead and merge.

[shlff]

Thanks @mmcky . I've pushed some modifications to the instructions. Now the PR is ready for merging.

I will also create a PR in repo lecture-python-intro to update the URL for the related lectures which use these datasets.