Broken pipe error

[curquiza]

Description

set -x MEILI_HOST_URL '***';
set -x MEILI_API_KEY '***';
set -x MEILI_INDEX_UID 'movies';
set -x MEILI_DATASET_PATH './huge_movies.json'

Dataset: it's a movie dataset with 477300 movies. The total size is 317M. Here is the kind of documents inside the dataset

  {
    "title": "Neither bird nor beast",
    "director": "",
    "producer": "",
    "tagline": "Based on a Mongolian tale.",
    "genres": [
      "Animation"
    ],
    "id": 444542,
    "overview": "Based on a mongolian tale.",
    "cast": "",
    "production_companies": [],
    "vote_count": 0,
    "vote_average": 0,
    "poster_path": "https://image.tmdb.org/t/p/w500/cKAubGWWR3feRw2VhtsknQNh9y5.jpg",
    "popularity": 0.6,
    "release_date": 445651200
}

I've sent the dataset by email @brunoocasali

Using this script:

require 'meilisearch'

API_KEY = ENV['MEILI_API_KEY']
HOST_URL = ENV['MEILI_HOST_URL']
INDEX_UID = ENV['MEILI_INDEX_UID']
DATASET_PATH =ENV['MEILI_DATASET_PATH']

client = MeiliSearch::Client.new(HOST_URL, API_KEY, timeout: 20)
index = client.index(INDEX_UID)

file = File.read(DATASET_PATH)
documents = JSON.parse(file)

documents.each_slice(500000) do |slice|
  task_status = index.add_documents(slice)
  puts "task uid: #{task_status['uid']}"
end

puts 'done!'

I get a broken pipe error

Traceback (most recent call last):
	28: from app.rb:17:in `<main>'
	27: from app.rb:17:in `each_slice'
	26: from app.rb:17:in `each'
	25: from app.rb:18:in `block in <main>'
	24: from /Users/curquiza/.rvm/gems/ruby-2.6.6/gems/meilisearch-0.18.2/lib/meilisearch/index.rb:71:in `add_documents'
	23: from /Users/curquiza/.rvm/gems/ruby-2.6.6/gems/meilisearch-0.18.2/lib/meilisearch/http_request.rb:38:in `http_post'
	22: from /Users/curquiza/.rvm/gems/ruby-2.6.6/gems/meilisearch-0.18.2/lib/meilisearch/http_request.rb:105:in `send_request'
	21: from /Users/curquiza/.rvm/gems/ruby-2.6.6/gems/meilisearch-0.18.2/lib/meilisearch/http_request.rb:39:in `block in http_post'
	20: from /Users/curquiza/.rvm/gems/ruby-2.6.6/gems/httparty-0.20.0/lib/httparty.rb:543:in `post'
	19: from /Users/curquiza/.rvm/gems/ruby-2.6.6/gems/httparty-0.20.0/lib/httparty.rb:613:in `perform_request'
	18: from /Users/curquiza/.rvm/gems/ruby-2.6.6/gems/httparty-0.20.0/lib/httparty/request.rb:152:in `perform'
	17: from /Users/curquiza/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/http.rb:1470:in `request'
	16: from /Users/curquiza/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/http.rb:920:in `start'
	15: from /Users/curquiza/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/http.rb:1472:in `block in request'
	14: from /Users/curquiza/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/http.rb:1479:in `request'
	13: from /Users/curquiza/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
	12: from /Users/curquiza/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/http.rb:1506:in `catch'
	11: from /Users/curquiza/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
	10: from /Users/curquiza/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
	 9: from /Users/curquiza/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
	 8: from /Users/curquiza/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/protocol.rb:247:in `write'
	 7: from /Users/curquiza/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
	 6: from /Users/curquiza/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
	 5: from /Users/curquiza/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
	 4: from /Users/curquiza/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
	 3: from /Users/curquiza/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/protocol.rb:275:in `each'
	 2: from /Users/curquiza/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/net/protocol.rb:277:in `block in write0'
	 1: from /Users/curquiza/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/openssl/buffering.rb:383:in `write_nonblock'
/Users/curquiza/.rvm/rubies/ruby-2.6.6/lib/ruby/2.6.0/openssl/buffering.rb:383:in `syswrite_nonblock': Broken pipe (Errno::EPIPE)

I agree the slice of 500000 is useless since the dataset has less than 500 000 documents

Expected behavior
Get the Meilisearch error saying the payload is too large?

Environment (please complete the following information):

• OS: MacOs (not M1)
• MeiliSearch version: v0.26.1
• meilisearch-ruby version: 0.18.2

[brunoocasali]

So, I got some ideas about this @curquiza I took some time to investigate it:

When the payload is too large, the Meilisearch server responds with a 413 status code (already expected). And the ruby gem is not capable of handling this connection interruption, so we need to fix that to deal with this error.

Another issue occurs when the user sets MEILI_HTTP_PAYLOAD_SIZE_LIMIT in order to allow a bigger payload, but the gem responds with a Net::ReadTimeout with #<TCPSocket:(closed)>.

To fulfill this issue I think we should fix both errors and respond with human-readable errors.

When ruby raise a Net::ReadTimeout responds with: MeiliSearch::TimeoutError
When ruby raises an Errno::EPIPE responds with MeiliSearch::CommunicationError.

[brunoocasali]

Just some more information, I think this kind of pipe error is directed related to the HTTParty library based on this comment from the http.rb gem.

irb(main):016:0> data = File.open("huge_movies.json").read;nil
=> nil
irb(main):017:0> HTTP.post("http://meilisearch:7700/indexes/movies/documents", body: data, headers: headers)
=> #<HTTP::Response/1.1 413 Payload Too Large {"content-length"=>"173", "connection"=>"close", "vary"=>"Origin, Access-Control-Request-Method, Access-Control-Request-Headers", "access-control-allow-origin"=>"*", "content-type"=>"application/json", "date"=>"Thu, 31 Mar 2022 21:17:10 GMT"}>
irb(main):019:0> HTTParty.post("http://meilisearch:7700/indexes/movies/documents", body: data, headers: headers)
Traceback (most recent call last):
       16: from /usr/local/lib/ruby/2.6.0/net/http.rb:1472:in `block in request'
       15: from /usr/local/lib/ruby/2.6.0/net/http.rb:1479:in `request'
       14: from /usr/local/lib/ruby/2.6.0/net/http.rb:1506:in `transport_request'
       13: from /usr/local/lib/ruby/2.6.0/net/http.rb:1506:in `catch'
       12: from /usr/local/lib/ruby/2.6.0/net/http.rb:1507:in `block in transport_request'
       11: from /usr/local/lib/ruby/2.6.0/net/http/generic_request.rb:123:in `exec'
       10: from /usr/local/lib/ruby/2.6.0/net/http/generic_request.rb:189:in `send_request_with_body'
        9: from /usr/local/lib/ruby/2.6.0/net/protocol.rb:247:in `write'
        8: from /usr/local/lib/ruby/2.6.0/net/protocol.rb:265:in `writing'
        7: from /usr/local/lib/ruby/2.6.0/net/protocol.rb:248:in `block in write'
        6: from /usr/local/lib/ruby/2.6.0/net/protocol.rb:275:in `write0'
        5: from /usr/local/lib/ruby/2.6.0/net/protocol.rb:275:in `each_with_index'
        4: from /usr/local/lib/ruby/2.6.0/net/protocol.rb:275:in `each'
        3: from /usr/local/lib/ruby/2.6.0/net/protocol.rb:277:in `block in write0'
        2: from /usr/local/lib/ruby/2.6.0/socket.rb:456:in `write_nonblock'
        1: from /usr/local/lib/ruby/2.6.0/socket.rb:456:in `__write_nonblock'
Errno::EPIPE (Broken pipe)
irb(main):021:0>