Storage+BigQuery.import notFound errors

[jakeorr]

I am uploading files to Google Cloud Storage and then importing them to Big Query and am seeing an intermittent notFound error during the import job.

Here is a simplified version of how I am streaming files to GCS:

rs.pipe(bucket.file('[filename]').createWriteStream({
  metadata: {
    contentType: 'text/csv',
    metadata: {
      custom: 'metadata'
    }
}).on('finish', function () {});

After file streaming has finished, the next step is importing to Big Query:

bigQuery
  .dataset([datasetId])
  .table([tableId])
  .import([bucket.file('[filename]'], {
    createDisposition: 'CREATE_IF_NEEDED',
    writeDisposition: 'WRITE_TRUNCATE',
    sourceFormat: 'CSV',
    schema: {
      fields: [fields array]
    }
  }, function (err, bqJob) {
    // Executes successfully
  });

I am seeing the error in the response I get from job.getMetadata():

bigQuery.job(bqJob.id).getMetadata(function (err, response) {});

response.status.errorResult.message: Not found: URI gs://[bucket]/[filename]
response.status.errorResult.reason: notFound
response.status.errorResult.errors: [{"reason":"notFound","message":"Not found: URI gs://[bucket]/[filename]"}]

After receiving this error, I have confirmed that the file in question does exist in GCS. Can there be some lag between when a stream has finished and when the file is actually available? If so is there a better way to know when the file(s) are ready for import into big query?

To be clear, this flow of events usually completes successfully, but I am intermittently seeing the notFound error. I would like to track it down or at least find a way to avoid it. Is this a known issue? (I haven't been able to find mention of it yet). Is it possible that rate limiting it occurring and causing this error to be generated?

[stephenplusplus]

Thanks for the detailed report. I'm putting together a sample script to try to reproduce.

As a quick thing that likely doesn't relate, I believe you can make the following change to your code:

.import([bucket.file('[filename]'], {
-   metadata: {
-     createDisposition: 'CREATE_IF_NEEDED',
      writeDisposition: 'WRITE_TRUNCATE',
-     sourceFormat: 'CSV',
      schema: {
        fields: [fields array]
      }
-   }
  }

The metadata property can be removed, and the other values I removed are also the default values.

I'll report back after I have a chance to try to reproduce.

[jakeorr]

Ah yeah, my mistake on including the metadata: {} wrapper. I have not been using it that way, just a copy paste+simplification mistake. I will update my comment. Thanks for the suggestions.

[stephenplusplus]

According to this doc, once a file upload is complete, it is immediately available. That would mean the only way BigQuery would get a 404 is if you received a "false finish" from file.createWriteStream, or somewhere else in the stack jumped the gun.

I put together a sample repo I've been using to try to reproduce. So far, no errors. Can you check this out to see where it might differ from your usage? Feel free to PR to make it more likely to fail.

[jakeorr]

Thanks for your efforts on this, Stephen. I just solved a bug in my code that was causing the import to happen before all files were finished streaming. From what I was seeing it's possible that the notFound error was happening from something unrelated, but I haven't been able to reproduce it. I will reopen if I run into the issue again. Thanks again!

[stephenplusplus]

Phew! No problem, glad you found the 🐛