spark: supplement spark-queries.md doc #3482

KnightChess · 2021-11-06T14:10:20Z

#3465
@jackye1995

jackye1995 · 2021-11-06T18:29:07Z

site/docs/spark-queries.md

+### All Data Files
+
+To show a table's valid data files and each file's metadata, run:
+> tip: a valid data file is one that is readable from any snapshot currently tracked by the table.


notes in the doc are in the form of !!! Note, example: https://github.com/apache/iceberg/blob/master/site/docs/spark-writes.md?plain=1#L298-L301

Also we should add the warning that this table may return duplicate rows.

jackye1995 · 2021-11-06T18:29:51Z

site/docs/spark-queries.md

+### All Manifests
+
+To show a table's valid file manifests and each file's metadata, run:
+> tip: a valid manifest file is one that is referenced from any snapshot currently tracked by the table.


similar comment as above, this table may return duplicate rows.

jackye1995 · 2021-11-06T18:30:21Z

site/docs/spark-queries.md

+### All Entries
+
+To show a table's valid manifest entries as rows, for both delete and data files, run:
+> tip: a valid manifest file is one that is referenced from any snapshot currently tracked by the table.


similar comment as above, this table may return duplicate rows.

KnightChess · 2021-11-07T02:49:30Z

I have modify it

rdblue · 2021-11-07T18:22:11Z

site/docs/spark-queries.md

 ```

-Note: 
+!!! Note


Doesn't this create an empty note?

yeah, I will resole it like this:

I don't think this should be in a note callout block. The previous formatting was just a paragraph, "Note:" with a list is better.

@KnightChess, can you fix this?

rdblue · 2021-11-07T18:22:45Z

site/docs/spark-queries.md

+    a valid manifest file is one that is referenced from any snapshot currently tracked by the table.
+
+!!! WARNING
+    this table may return **duplicate** rows


Documentation should use correct capitalization. In this case, sentences should start with a capital letter.

Looks like there are several cases of sentences that don't start with capitals, so be sure to fix all of them, please!

rdblue · 2021-11-07T18:24:20Z

site/docs/spark-queries.md

+----------------+------------+----------+
+```
+
+### Entries


While we support this, I'm not sure that we want this in the standard documentation for Spark users. The entries table will show all entries in manifests, for both deleted and live files in the table. I think interpreting this table is more of an advanced use.

What do you think @samredai?

+1 I totally agree. I can't see a reason for a typical user to use this table and having it in this section of the docs could be a signal that they should be using it. There's also an instinct of data engineers coming from Hive tables to want to find the location of the data files and start messing around there so it's beneficial if we strengthen that abstraction by example in the docs (maybe even explicitly call it out)

Sorry I am late. I personally use this and have pointed others to build on this, because it's currently the only way to track when is the last time data is added, or how much data is added, to a partition.

But that being said, do understand it's a bit too complicated to document in the standard guide here, though just wondering what way advanced users can ever figure this out.

rdblue · 2021-11-07T18:26:03Z

site/docs/spark-queries.md

 +-------------------------------------------------------------------------+-------------+--------------+--------------------+--------------------+------------------+-------------------+------------------+-----------------+-----------------+--------------+---------------+
 ```

+### All Data Files


I think it makes sense to have a separate section for the "all" tables. These aren't needed nearly as often and require warnings, as you have in these sections.

rdblue · 2021-11-07T18:26:47Z

site/docs/spark-queries.md


+### All Data Files
+
+To show a table's valid data files and each file's metadata, run:


I don't think it is a good idea to end a paragraph with "run:" and then put two callouts between it and the SQL that it is referencing.

rdblue · 2021-11-07T18:28:18Z

site/docs/spark-queries.md

+|      0|s3:.../dt=20210102/xxx.parquet|    PARQUET|{20210102}|          14|              2444|{1 -> 94, 2 -> 17}|{1 -> 14, 2 -> 14}|  {1 -> 0, 2 -> 0}|              {}|{1 -> 1, 2 -> 20210102}|{1 -> 2, 2 -> 20210102}|        null|          [4]|        null|            0|
+|      0|s3:.../dt=20210103/xxx.parquet|    PARQUET|{20210103}|          14|              2444|{1 -> 94, 2 -> 17}|{1 -> 14, 2 -> 14}|  {1 -> 0, 2 -> 0}|              {}|{1 -> 1, 2 -> 20210103}|{1 -> 3, 2 -> 20210103}|        null|          [4]|        null|            0|
+|      0|s3:.../dt=20210104/xxx.parquet|    PARQUET|{20210104}|          14|              2444|{1 -> 94, 2 -> 17}|{1 -> 14, 2 -> 14}|  {1 -> 0, 2 -> 0}|              {}|{1 -> 1, 2 -> 20210104}|{1 -> 3, 2 -> 20210104}|        null|          [4]|        null|            0|
+-------+------------------------------+-----------+----------+------------+------------------+------------------+------------------+------------------+----------------+-----------------------+-----------------------+------------+-------------+------------+-------------+


Can you fix the formatting of this table? You can see my recommendations on how to get a nicely formatted table here: #3422 (comment)

KnightChess · 2021-11-08T12:42:35Z

@rdblue

I have update all table format
use three-level title all metadata to split it
other fix

rdblue · 2021-11-08T16:30:29Z

site/docs/spark-queries.md

+    The metadata is readable from any snapshot currently tracked by the table
+
+!!! WARNING
+    The table may return **duplicate** rows


What is "the table"?

my mistake, the table's metadata

KnightChess · 2021-11-10T05:13:52Z

site/docs/spark-queries.md

 SELECT * FROM prod.db.table.history
 ```
-```text
-+-------------------------+---------------------+---------------------+---------------------+
-| made_current_at         | snapshot_id         | parent_id           | is_current_ancestor |
-+-------------------------+---------------------+---------------------+---------------------+
+
+<div class="markdown-table-container" markdown="block">
+| made_current_at | snapshot_id  | parent_id | is_current_ancestor |
+| -- | -- | -- | -- |
 | 2019-02-08 03:29:51.215 | 5781947118336215154 | NULL                | true                |
 | 2019-02-08 03:47:55.948 | 5179299526185056830 | 5781947118336215154 | true                |
 | 2019-02-09 16:24:30.13  | 296410040247533544  | 5179299526185056830 | false               |
 | 2019-02-09 16:32:47.336 | 2999875608062437330 | 5179299526185056830 | true                |
 | 2019-02-09 19:42:03.919 | 8924558786060583479 | 2999875608062437330 | true                |
 | 2019-02-09 19:49:16.343 | 6536733823181975045 | 8924558786060583479 | true                |
-+-------------------------+---------------------+---------------------+---------------------+
-```
+</div>


@szehon-ho here, and I use the css test in my local branch

.markdown-table-container { width: 780px; overflow-x: scroll; }

Thanks for pointing this out @KnightChess ! If we're going to use the container class for all markdown tables, we can use overflow-x: auto; instead and it will only show the bar when there's overflow.

To prevent a merge conflict, I'll mention this over in PR #3422 so the css can be updated there.

site/docs/spark-queries.md

jackye1995 · 2021-11-22T21:06:54Z

site/docs/spark-queries.md

+</div>
+
+!!! Note
+    Diffrent from metadata table [files](#files), This table exposes internal details, like files that have been deleted.


nit: this table (lower case)

jackye1995 · 2021-11-22T21:08:22Z

site/docs/spark-queries.md

+</div>
+
+!!! Note
+    Like metadata table [entries](#entries), contain delete data files too, different from [all_data_files](#all-data-files)


this sentence sounds a bit awkward, what about:

Like metadata table [entries](#entries), this metadata table is different from [all_data_files](#all-data-files) and contains deleted data files.

jackye1995 · 2021-12-06T21:17:59Z

@KnightChess any updates?

KnightChess · 2021-12-07T01:28:45Z

@KnightChess any updates?

sorry, my fault. There was no change in the email I received so I did not confirm on github. I will do it today

KnightChess · 2021-12-07T01:31:04Z

@jackye1995 cc

jackye1995

thanks for the quick update, looks good to me

site/docs/spark-queries.md

rdblue · 2021-12-09T00:26:58Z

site/docs/spark-queries.md

+!!! Note
+    Like metadata table [entries](#entries), this metadata table is different from [all_data_files](#all-data-files) and contains deleted data files.
+
+!!! WARNING


This should be "Warning" rather than all caps.

site/docs/spark-queries.md

KnightChess · 2021-12-24T08:21:36Z

update and fix conflict from #3422 , the style for markdown table look good.
cc: @rdblue @samredai

site/docs/spark-queries.md

rdblue · 2021-12-29T21:58:48Z

site/docs/spark-queries.md

+### All Metadata
+
+!!! Note
+    The metadata is readable from any snapshot currently tracked by the table


This doesn't make sense to me.

rdblue · 2021-12-29T22:00:01Z

site/docs/spark-queries.md

+|  {20211002, 10}|           1|         1|
+</div>
+
+### All Metadata


I think this should be "All Metadata Tables" (note plural tables)

And this should have a paragraph explaining that these tables are unions of the metadata tables that are specific to a snapshot.

site/docs/spark-queries.md

szehon-ho · 2022-03-18T07:27:14Z

site/docs/spark-queries.md

+----------------+------------+----------+
+```
+
+### Entries


Sorry I am late. I personally use this and have pointed others to build on this, because it's currently the only way to track when is the last time data is added, or how much data is added, to a partition.

But that being said, do understand it's a bit too complicated to document in the standard guide here, though just wondering what way advanced users can ever figure this out.

szehon-ho

Hi @KnightChess , if you still have time, could you rebase and address the comments? I would like to move forward on it, I think it's a valuable change.

ajantha-bhat · 2022-03-24T03:01:27Z

Hi @KnightChess , if you still have time, could you rebase and address the comments? I would like to move forward on it, I think it's a valuable change.

Agree, I thought of adding this doc few months ago, but because this PR was already there, I was waiting for it to getting merged.

@KnightChess : If you don't have time, you can let us know. I can add you as a coauthor and take this PR forward.

KnightChess · 2022-03-24T03:18:36Z

@szehon-ho @ajantha-bhat sorry for the late reply, I will take this PR forward in the next few days.

ajantha-bhat · 2022-03-24T03:40:56Z

@szehon-ho @ajantha-bhat sorry for the late reply, I will take this PR forward in the next few days.

Awesome. Looking forward to it.

szehon-ho

Thanks @KnightChess for continuing it, replied to some of the comments, and look forward to the changes. I'll be out next week, but can look if i have time, or after that.

site/docs/spark-queries.md

KnightChess · 2022-03-26T10:02:30Z

@szehon-ho update other except Advanced Usage, and adapt new doc styple for hugo

szehon-ho

Sorry for delay, I just got back from vacation. Thanks @KnightChess for rebase/fix, and others for confirming the earlier comment.

I made another round of review and some smaller comments, can you please take a look? The change is getting close, and should be good soon. Thanks again.

docs/spark/spark-queries.md

szehon-ho

Thanks @KnightChess I built and found just one broken link in the new doc, should be good after this.

docs/spark/spark-queries.md

szehon-ho · 2022-04-08T16:55:53Z

Merged, thanks @KnightChess for your patience and contribution

KnightChess · 2022-04-09T00:45:22Z

thanks @szehon-ho @rdblue @jackye1995 @samredai a lot for instruct patiently

samredai · 2022-04-09T17:15:33Z

Thanks @KnightChess!

github-actions bot added the docs label Nov 6, 2021

jackye1995 reviewed Nov 6, 2021

View reviewed changes

rdblue reviewed Nov 7, 2021

View reviewed changes

KnightChess mentioned this pull request Nov 8, 2021

Doc: Update Files metadata table #3422

Merged

rdblue reviewed Nov 8, 2021

View reviewed changes

KnightChess force-pushed the doc-for-system-tables branch from b5e6fc2 to 369f55c Compare November 10, 2021 05:04

KnightChess commented Nov 10, 2021

View reviewed changes

KnightChess commented Nov 11, 2021

View reviewed changes

site/docs/spark-queries.md Outdated Show resolved Hide resolved

KnightChess force-pushed the doc-for-system-tables branch from 19e3035 to f0b198c Compare November 17, 2021 02:46

jackye1995 added this to the Iceberg 0.13.0 Release milestone Nov 18, 2021

jackye1995 reviewed Nov 22, 2021

View reviewed changes

jackye1995 approved these changes Dec 7, 2021

View reviewed changes

jackye1995 requested review from rdblue and samredai December 7, 2021 02:34

rdblue reviewed Dec 9, 2021

View reviewed changes

site/docs/spark-queries.md Outdated Show resolved Hide resolved

rdblue reviewed Dec 9, 2021

View reviewed changes

site/docs/spark-queries.md Outdated Show resolved Hide resolved

KnightChess force-pushed the doc-for-system-tables branch from 3814a2b to aa6570b Compare December 24, 2021 08:12

samredai reviewed Dec 28, 2021

View reviewed changes

site/docs/spark-queries.md Outdated Show resolved Hide resolved

rdblue reviewed Dec 29, 2021

View reviewed changes

site/docs/spark-queries.md Outdated Show resolved Hide resolved

szehon-ho reviewed Mar 18, 2022

View reviewed changes

spark: supplement spark-queries.md doc

2309c74

szehon-ho reviewed Mar 25, 2022

View reviewed changes

site/docs/spark-queries.md Outdated Show resolved Hide resolved

site/docs/spark-queries.md Outdated Show resolved Hide resolved

spark: update spark-queries according to advice

998f271

KnightChess force-pushed the doc-for-system-tables branch from b821ae0 to 998f271 Compare March 26, 2022 09:57

spark: spark-queries remove advanced usage

c3f0555

szehon-ho reviewed Apr 4, 2022

View reviewed changes

KnightChess added 2 commits April 7, 2022 10:02

spark: update nit

3e36a56

spark: update nit

be553ce

szehon-ho reviewed Apr 7, 2022

View reviewed changes

docs/spark/spark-queries.md Outdated Show resolved Hide resolved

spark: update nit

fab949a

szehon-ho approved these changes Apr 8, 2022

View reviewed changes

szehon-ho merged commit 49dad0c into apache:master Apr 8, 2022


		### All Data Files

		To show a table's valid data files and each file's metadata, run:

spark: supplement spark-queries.md doc #3482

spark: supplement spark-queries.md doc #3482

Uh oh!

Conversation

KnightChess commented Nov 6, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KnightChess commented Nov 7, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KnightChess commented Nov 8, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samredai Nov 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jackye1995 Nov 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jackye1995 commented Dec 6, 2021

Uh oh!

KnightChess commented Dec 7, 2021

Uh oh!

KnightChess commented Dec 7, 2021

Uh oh!

jackye1995 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

KnightChess commented Dec 24, 2021

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

samredai Nov 10, 2021 •

edited

Loading

jackye1995 Nov 22, 2021 •

edited

Loading

szehon-ho left a comment •

edited

Loading

szehon-ho left a comment •

edited

Loading