fix: parsing timestamp with specified precision #153

bednar · 2024-07-09T12:31:02Z

Closes #154

Proposed Changes

This PR modifies how the client parses timestamps. Recently, InfluxDB 3 began returning timestamps in nanoseconds, necessitating a change in the parsing method. The parsing is now more robust.

https://app.circleci.com/pipelines/github/InfluxCommunity/influxdb3-java/510/workflows/51f380f3-1cd0-49bd-bb11-fe2f72cb34dc/jobs/3031?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link&utm_content=summary

Checklist

codecov · 2024-07-09T12:35:44Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.19%. Comparing base (8892141) to head (4bafab9).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #153      +/-   ##
==========================================
+ Coverage   89.03%   89.19%   +0.16%     
==========================================
  Files          16       16              
  Lines         866      879      +13     
  Branches      131      133       +2     
==========================================
+ Hits          771      784      +13     
  Misses         40       40              
  Partials       55       55

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

karel-rehor

Tests pass locally with openjdk 20.0.2. Changes make sense.

sahil-devinci · 2024-07-16T09:58:55Z

Are you sure it will work? I copied the above code and converted the timestamp as per above code, year is all wrong.

pointValue timestamp: 1721051820455000000000000

long nanoseconds = TimeUnit.NANOSECONDS.convert((Long) dateTime.longValue(), TimeUnit.NANOSECONDS);
		Instant.ofEpochSecond(0, nanoseconds);

2207-05-30T09:54:06.253330432Z

karel-rehor · 2024-07-16T12:53:49Z

@sahil-devinci

I'm curious where you got that timestamp as it would be base femto or 10^-15.

e.g.

1721134115000000000 - nano
1721051820455000000000000 - femto

Just to be sure I've repeated the implementation in local test code and everything looks correct.

e.g.

  Instant now = Instant.now();
  System.out.println("now " + now);
  ...
  Long seconds = now.getEpochSecond();
  long nsFromS = java.util.concurrent.TimeUnit.NANOSECONDS.convert(seconds, java.util.concurrent.TimeUnit.SECONDS);
  System.out.println("DEBUG nsFromS " + nsFromS);
  System.out.println("DEBUG nsFromS as instant " + Instant.ofEpochSecond(0, nsFromS));

From the console:

...
DEBUG nsFromS 1721134115000000000
DEBUG nsFromS as instant 2024-07-16T12:48:35Z
...

I also checked the value in your comment.

  // reduced to nano
    long nsFromX = java.util.concurrent.TimeUnit.NANOSECONDS.convert(1_721_051_820_455_000_000L,
      java.util.concurrent.TimeUnit.NANOSECONDS);
    System.out.println("DEBUG nsFromX " + nsFromX);
    System.out.println("DEBUG nsFromX as instant " + Instant.ofEpochSecond(0, nsFromX));
  // original
    BigInteger bi = new BigInteger("1721051820455000000000000");
    long nsFromX2 = java.util.concurrent.TimeUnit.NANOSECONDS.convert(
      bi.longValue(),
      java.util.concurrent.TimeUnit.NANOSECONDS);
    System.out.println("DEBUG nsFromX2 " + nsFromX2);
    System.out.println("DEBUG nsFromX2 as instant " + Instant.ofEpochSecond(0, nsFromX2));

If you reduce the value back to nano from femto you get a timestamp from Monday July 15 2024.

Some commented results:

1721133051000000000 // base nano 10^-9 => 2024-07-16T12:30:51Z
1721051820455000000 // ^^^ Ditto ^^^^^ => 2024-07-15T13:57:00.455Z

1721051820455000000000000 // base femto 10^-15  over flows MAX_LONG at least once resulting in 
7491866046253330432 // base nano 10^-9 => 2207-05-30T09:54:06.253330432Z

sahil-devinci · 2024-07-16T22:39:43Z

@sahil-devinci

I'm curious where you got that timestamp as it would be base femto or 10^-15.

e.g.

1721134115000000000 - nano
1721051820455000000000000 - femto

Just to be sure I've repeated the implementation in local test code and everything looks correct.

e.g.

  Instant now = Instant.now();
  System.out.println("now " + now);
  ...
  Long seconds = now.getEpochSecond();
  long nsFromS = java.util.concurrent.TimeUnit.NANOSECONDS.convert(seconds, java.util.concurrent.TimeUnit.SECONDS);
  System.out.println("DEBUG nsFromS " + nsFromS);
  System.out.println("DEBUG nsFromS as instant " + Instant.ofEpochSecond(0, nsFromS));

From the console:

...
DEBUG nsFromS 1721134115000000000
DEBUG nsFromS as instant 2024-07-16T12:48:35Z
...

I also checked the value in your comment.

  // reduced to nano
    long nsFromX = java.util.concurrent.TimeUnit.NANOSECONDS.convert(1_721_051_820_455_000_000L,
      java.util.concurrent.TimeUnit.NANOSECONDS);
    System.out.println("DEBUG nsFromX " + nsFromX);
    System.out.println("DEBUG nsFromX as instant " + Instant.ofEpochSecond(0, nsFromX));
  // original
    BigInteger bi = new BigInteger("1721051820455000000000000");
    long nsFromX2 = java.util.concurrent.TimeUnit.NANOSECONDS.convert(
      bi.longValue(),
      java.util.concurrent.TimeUnit.NANOSECONDS);
    System.out.println("DEBUG nsFromX2 " + nsFromX2);
    System.out.println("DEBUG nsFromX2 as instant " + Instant.ofEpochSecond(0, nsFromX2));

If you reduce the value back to nano from femto you get a timestamp from Monday July 15 2024.

Some commented results:

1721133051000000000 // base nano 10^-9 => 2024-07-16T12:30:51Z
1721051820455000000 // ^^^ Ditto ^^^^^ => 2024-07-15T13:57:00.455Z

1721051820455000000000000 // base femto 10^-15  over flows MAX_LONG at least once resulting in 
7491866046253330432 // base nano 10^-9 => 2207-05-30T09:54:06.253330432Z

I am getting it from pointValue.getTimestamp. I am using version 0.8.0 of this library.

`try (Stream stream = influxDBClientThreePhase.queryPoints(formattedQuery)) {
stream.forEach((PointValues p) -> extractPoints(queryHealthTrendList, p, localZoneId));
}

public List<ThreePhaseHealthTrend> extractPoints(List<ThreePhaseHealthTrend> list, PointValues p, ZoneId zoneId) {
	ThreePhaseHealthTrend metrics = new ThreePhaseHealthTrend();
	// Power
	metrics.setPA(p.getField("actPowerA", Double.class));
	metrics.setPB(p.getField("actPowerB", Double.class));
	metrics.setPC(p.getField("actPowerC", Double.class));
	// Current
	metrics.setCA(p.getField("currentA", Double.class));
	metrics.setCB(p.getField("currentB", Double.class));
	metrics.setCC(p.getField("currentC", Double.class));

	// PF
	metrics.setPfA(p.getField("pfA", Double.class));
	metrics.setPfB(p.getField("pfB", Double.class));
	metrics.setPfC(p.getField("pfC", Double.class));
	// Voltage
	metrics.setVA(p.getField("voltageA", Double.class));
	metrics.setVB(p.getField("voltageB", Double.class));
	metrics.setVC(p.getField("voltageC", Double.class));

	metrics.setTime(DateTimeUtils.fromInfluxInstantToSecond(**p.getTimestamp()**, zoneId));			
			`

p.getTimestamp() is the value I shared above. And when I run the query directly in influx db serverless it shows correct value.

bednar · 2024-07-17T05:29:36Z

Hi @sahil-devinci,

thanks for insight in this issue.

Are you sure it will work? I copied the above code and converted the timestamp as per above code, year is all wrong.

pointValue timestamp: 1721051820455000000000000
long nanoseconds = TimeUnit.NANOSECONDS.convert((Long) dateTime.longValue(), TimeUnit.NANOSECONDS);
		Instant.ofEpochSecond(0, nanoseconds);
2207-05-30T09:54:06.253330432Z

What type is the dateTime?

I am getting it from pointValue.getTimestamp. I am using version 0.8.0 of this library.

`try (Stream stream = influxDBClientThreePhase.queryPoints(formattedQuery)) { stream.forEach((PointValues p) -> extractPoints(queryHealthTrendList, p, localZoneId)); }

public List<ThreePhaseHealthTrend> extractPoints(List<ThreePhaseHealthTrend> list, PointValues p, ZoneId zoneId) {
	ThreePhaseHealthTrend metrics = new ThreePhaseHealthTrend();
	// Power
	metrics.setPA(p.getField("actPowerA", Double.class));
	metrics.setPB(p.getField("actPowerB", Double.class));
	metrics.setPC(p.getField("actPowerC", Double.class));
	// Current
	metrics.setCA(p.getField("currentA", Double.class));
	metrics.setCB(p.getField("currentB", Double.class));
	metrics.setCC(p.getField("currentC", Double.class));

	// PF
	metrics.setPfA(p.getField("pfA", Double.class));
	metrics.setPfB(p.getField("pfB", Double.class));
	metrics.setPfC(p.getField("pfC", Double.class));
	// Voltage
	metrics.setVA(p.getField("voltageA", Double.class));
	metrics.setVB(p.getField("voltageB", Double.class));
	metrics.setVC(p.getField("voltageC", Double.class));

	metrics.setTime(DateTimeUtils.fromInfluxInstantToSecond(**p.getTimestamp()**, zoneId));			
			`

p.getTimestamp() is the value I shared above. And when I run the query directly in influx db serverless it shows correct value.

The version 0.8.0 has an issue which is addressed by this PR. The com.influxdb.v3.client.PointValues#getTimestamp should always return nanoseconds precision.

Can you please test your code with the fixed version of the client from this PR?

sahil-devinci · 2024-07-17T06:18:59Z

I am using 0.8.0 . And datetime is of type Number.

…

On Wed, Jul 17, 2024, 5:29 PM Jakub Bednář ***@***.***> wrote: Hi @sahil-devinci <https://github.com/sahil-devinci>, thanks for insight in this issue. Are you sure it will work? I copied the above code and converted the timestamp as per above code, year is all wrong. pointValue timestamp: 1721051820455000000000000 long nanoseconds = TimeUnit.NANOSECONDS.convert((Long) dateTime.longValue(), TimeUnit.NANOSECONDS); Instant.ofEpochSecond(0, nanoseconds); *2207-05-30T09:54:06.253330432Z* What type is the dateTime? I am getting it from pointValue.getTimestamp. I am using version 0.8.0 of this library. `try (Stream stream = influxDBClientThreePhase.queryPoints(formattedQuery)) { stream.forEach((PointValues p) -> extractPoints(queryHealthTrendList, p, localZoneId)); } public List<ThreePhaseHealthTrend> extractPoints(List<ThreePhaseHealthTrend> list, PointValues p, ZoneId zoneId) { ThreePhaseHealthTrend metrics = new ThreePhaseHealthTrend(); // Power metrics.setPA(p.getField("actPowerA", Double.class)); metrics.setPB(p.getField("actPowerB", Double.class)); metrics.setPC(p.getField("actPowerC", Double.class)); // Current metrics.setCA(p.getField("currentA", Double.class)); metrics.setCB(p.getField("currentB", Double.class)); metrics.setCC(p.getField("currentC", Double.class)); // PF metrics.setPfA(p.getField("pfA", Double.class)); metrics.setPfB(p.getField("pfB", Double.class)); metrics.setPfC(p.getField("pfC", Double.class)); // Voltage metrics.setVA(p.getField("voltageA", Double.class)); metrics.setVB(p.getField("voltageB", Double.class)); metrics.setVC(p.getField("voltageC", Double.class)); metrics.setTime(DateTimeUtils.fromInfluxInstantToSecond(**p.getTimestamp()**, zoneId)); ` p.getTimestamp() is the value I shared above. And when I run the query directly in influx db serverless it shows correct value. The version 0.8.0 has an issue which is addressed by this PR. The com.influxdb.v3.client.PointValues#getTimestamp should always return nanoseconds precision. Can you please test your code with the fixed version of the client from this PR? — Reply to this email directly, view it on GitHub <#153 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2AABBZYWMBS7BHFUHLTR4DZMX6NNAVCNFSM6AAAAABKSYSRV6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZSGQ2DSMZWGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

bednar · 2024-07-22T13:21:29Z

@sahil-devinci the [com.influxdb.v3.client.PointValues#getTimestamp] should always return the timestamp in nanoseconds. I've updated the JavaDoc in this PR.

bednar added 2 commits July 9, 2024 14:28

fix: parsing timestamp with specified precision

2c325ec

fix: style

8b2370f

bednar marked this pull request as ready for review July 9, 2024 12:34

bednar requested a review from karel-rehor July 9, 2024 12:34

docs: update CHANGELOG.md

f804533

bednar mentioned this pull request Jul 15, 2024

Odd Timestamp Precision Behaviour #154

Closed

karel-rehor approved these changes Jul 16, 2024

View reviewed changes

alespour mentioned this pull request Jul 18, 2024

fix: add Edge authentication support #158

Merged

5 tasks

bednar added 2 commits July 22, 2024 15:18

chore: add test for TimeStamp without TimeZone - TimeStampNanoTZVector

8c0a2f2

docs: correct doc for nanoseconds

4bafab9

bednar merged commit 87cd880 into main Jul 22, 2024

bednar deleted the parsing-timestamp branch July 22, 2024 13:29

bednar added this to the 0.9.0 milestone Jul 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: parsing timestamp with specified precision #153

fix: parsing timestamp with specified precision #153

Uh oh!

bednar commented Jul 9, 2024 •

edited

Loading

Uh oh!

codecov bot commented Jul 9, 2024 •

edited

Loading

Uh oh!

karel-rehor left a comment

Uh oh!

sahil-devinci commented Jul 16, 2024

Uh oh!

karel-rehor commented Jul 16, 2024

Uh oh!

sahil-devinci commented Jul 16, 2024 •

edited

Loading

Uh oh!

bednar commented Jul 17, 2024

Uh oh!

sahil-devinci commented Jul 17, 2024 via email

Uh oh!

bednar commented Jul 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fix: parsing timestamp with specified precision #153

fix: parsing timestamp with specified precision #153

Uh oh!

Conversation

bednar commented Jul 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Changes

Checklist

Uh oh!

codecov bot commented Jul 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

karel-rehor left a comment

Choose a reason for hiding this comment

Uh oh!

sahil-devinci commented Jul 16, 2024

Uh oh!

karel-rehor commented Jul 16, 2024

Uh oh!

sahil-devinci commented Jul 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bednar commented Jul 17, 2024

Uh oh!

sahil-devinci commented Jul 17, 2024 via email

Uh oh!

bednar commented Jul 22, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bednar commented Jul 9, 2024 •

edited

Loading

codecov bot commented Jul 9, 2024 •

edited

Loading

sahil-devinci commented Jul 16, 2024 •

edited

Loading