Skip to content

✨ Support PARTIAL extension (RFC9394) #368

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
nevans wants to merge 3 commits into from
Closed

✨ Support PARTIAL extension (RFC9394) #368

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
10e19be
♻️📚 Minor `#fetch` refactor and doc updates
nevans Dec 20, 2024
48062ce
✨ Add support for `PARTIAL` extension (RFC9394) [🚧TODO: test #to_a]
nevans Dec 18, 2024
4de8e6b
✨ Add `partial` fetch modifier to uid_fetch
nevans Dec 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
134 changes: 103 additions & 31 deletions lib/net/imap.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -534,6 +534,11 @@ module Net
# See FetchData#emailid and FetchData#emailid.
# - Updates #status with support for the +MAILBOXID+ status attribute.
#
# ==== RFC9394: +PARTIAL+
# - Updates #search, #uid_search with the +PARTIAL+ return option which adds
# ESearchResult#partial return data.
# - Updates #uid_fetch with the +partial+ modifier.
#
# == References
#
# [{IMAP4rev1}[https://www.rfc-editor.org/rfc/rfc3501.html]]::
Expand Down Expand Up @@ -701,6 +706,11 @@ module Net
# Gondwana, B., Ed., "IMAP Extension for Object Identifiers",
# RFC 8474, DOI 10.17487/RFC8474, September 2018,
# <https://www.rfc-editor.org/info/rfc8474>.
# [PARTIAL[https://www.rfc-editor.org/info/rfc9394]]:
# Melnikov, A., Achuthan, A., Nagulakonda, V., and L. Alves,
# "IMAP PARTIAL Extension for Paged SEARCH and FETCH", RFC 9394,
# DOI 10.17487/RFC9394, June 2023,
# <https://www.rfc-editor.org/info/rfc9394>.
#
# === IANA registries
# * {IMAP Capabilities}[http://www.iana.org/assignments/imap4-capabilities]
Expand Down Expand Up @@ -1971,8 +1981,9 @@ def uid_expunge(uid_set)
# the server to return an ESearchResult instead of a SearchResult, but some
# servers disobey this requirement. <em>Requires an extended search
# capability, such as +ESEARCH+ or +IMAP4rev2+.</em>
# See {"Argument translation"}[rdoc-ref:#search@Argument+translation]
# and {"Return options"}[rdoc-ref:#search@Return+options], below.
# See {"Argument translation"}[rdoc-ref:#search@Argument+translation] and
# {"Supported return options"}[rdoc-ref:#search@Supported+return+options],
# below.
#
# +charset+ is the name of the {registered character
# set}[https://www.iana.org/assignments/character-sets/character-sets.xhtml]
Expand Down Expand Up @@ -2082,39 +2093,56 @@ def uid_expunge(uid_set)
# <em>*WARNING:* This is vulnerable to injection attacks when external
# inputs are used.</em>
#
# ==== Return options
# ==== Supported return options
#
# For full definitions of the standard return options and return data, see
# the relevant RFCs.
#
# ===== +ESEARCH+ or +IMAP4rev2+
#
# The following return options require either +ESEARCH+ or +IMAP4rev2+.
# See [{RFC4731 §3.1}[https://rfc-editor.org/rfc/rfc4731#section-3.1]] or
# [{IMAP4rev2 §6.4.4}[https://www.rfc-editor.org/rfc/rfc9051.html#section-6.4.4]].
#
# [+ALL+]
# Returns ESearchResult#all with a SequenceSet of all matching sequence
# numbers or UIDs. This is the default, when return options are empty.
#
# For compatibility with SearchResult, ESearchResult#to_a returns an
# Array of message sequence numbers or UIDs.
#
# <em>Requires either the +ESEARCH+ or +IMAP4rev2+ capabability.</em>
# {[RFC4731]}[https://rfc-editor.org/rfc/rfc4731]
# {[RFC9051]}[https://rfc-editor.org/rfc/rfc9051]
#
# [+COUNT+]
# Returns ESearchResult#count with the number of matching messages.
#
# <em>Requires either the +ESEARCH+ or +IMAP4rev2+ capabability.</em>
# {[RFC4731]}[https://rfc-editor.org/rfc/rfc4731]
# {[RFC9051]}[https://rfc-editor.org/rfc/rfc9051]
#
# [+MAX+]
# Returns ESearchResult#max with the highest matching sequence number or
# UID.
#
# <em>Requires either the +ESEARCH+ or +IMAP4rev2+ capabability.</em>
# {[RFC4731]}[https://rfc-editor.org/rfc/rfc4731]
# {[RFC9051]}[https://rfc-editor.org/rfc/rfc9051]
#
# [+MIN+]
# Returns ESearchResult#min with the lowest matching sequence number or
# UID.
#
# ===== +CONDSTORE+
# <em>Requires either the +ESEARCH+ or +IMAP4rev2+ capabability.</em>
# {[RFC4731]}[https://rfc-editor.org/rfc/rfc4731]
# {[RFC9051]}[https://rfc-editor.org/rfc/rfc9051]
#
# ESearchResult#modseq return data does not have a corresponding return
# option. Instead, it is returned if the +MODSEQ+ search key is used or
# when the +CONDSTORE+ extension is enabled for the selected mailbox.
# See [{RFC4731 §3.2}[https://www.rfc-editor.org/rfc/rfc4731#section-3.2]]
# or [{RFC7162 §2.1.5}[https://www.rfc-editor.org/rfc/rfc7162#section-3.1.5]].
# [+PARTIAL+ _range_]
# Returns ESearchResult#partial with a SequenceSet of a subset of
# matching sequence numbers or UIDs, as selected by _range_. As with
# sequence numbers, the first result is +1+: <tt>1..500</tt> selects the
# first 500 search results (in mailbox order), <tt>501..1000</tt> the
# second 500, and so on. _range_ may also be negative: <tt>-500..-1</tt>
# selects the last 500 search results.
#
# <em>Requires either the <tt>CONTEXT=SEARCH</tt> or +PARTIAL+ capabability.</em>
# {[RFC5267]}[https://rfc-editor.org/rfc/rfc5267]
# {[RFC9394]}[https://rfc-editor.org/rfc/rfc9394]
#
# ===== +RFC4466+ compatible extensions
#
Expand All @@ -2125,6 +2153,14 @@ def uid_expunge(uid_set)
# intentionally _unstable_ API. Future releases may return different
# (incompatible) objects, <em>without deprecation or warning</em>.
#
# ===== +MODSEQ+ return data
#
# ESearchResult#modseq return data does not have a corresponding return
# option. Instead, it is returned if the +MODSEQ+ search key is used or
# when the +CONDSTORE+ extension is enabled for the selected mailbox.
# See [{RFC4731 §3.2}[https://www.rfc-editor.org/rfc/rfc4731#section-3.2]]
# or [{RFC7162 §2.1.5}[https://www.rfc-editor.org/rfc/rfc7162#section-3.1.5]].
#
# ==== Search keys
#
# For full definitions of the standard search +criteria+,
Expand Down Expand Up @@ -2357,14 +2393,7 @@ def uid_search(...)
# Sends a {FETCH command [IMAP4rev1 §6.4.5]}[https://www.rfc-editor.org/rfc/rfc3501#section-6.4.5]
# to retrieve data associated with a message in the mailbox.
#
# The +set+ parameter is a number or a range between two numbers,
# or an array of those. The number is a message sequence number,
# where -1 represents a '*' for use in range notation like 100..-1
# being interpreted as '100:*'. Beware that the +exclude_end?+
# property of a Range object is ignored, and the contents of a
# range are independent of the order of the range endpoints as per
# the protocol specification, so 1...5, 5..1 and 5...1 are all
# equivalent to 1..5.
# The +set+ parameter must be a valid input to SequenceSet::[].
#
# +attr+ is a list of attributes to fetch; see the documentation
# for FetchData for a list of valid attributes.
Expand Down Expand Up @@ -2403,19 +2432,42 @@ def uid_search(...)
# {[RFC7162]}[https://tools.ietf.org/html/rfc7162] in order to use the
# +changedsince+ argument. Using +changedsince+ implicitly enables the
# +CONDSTORE+ extension.
def fetch(set, attr, mod = nil, changedsince: nil)
fetch_internal("FETCH", set, attr, mod, changedsince: changedsince)
def fetch(...)
fetch_internal("FETCH", ...)
end

# :call-seq:
# uid_fetch(set, attr, changedsince: nil) -> array of FetchData
# uid_fetch(set, attr, changedsince: nil, partial: nil) -> array of FetchData
#
# Sends a {UID FETCH command [IMAP4rev1 §6.4.8]}[https://www.rfc-editor.org/rfc/rfc3501#section-6.4.8]
# to retrieve data associated with a message in the mailbox.
#
# Similar to #fetch, but the +set+ parameter contains unique identifiers
# instead of message sequence numbers.
#
# When #uid_fetch may also be given a +partial+ range, which can be used to
# limit the number of results. <em>Requires the +PARTIAL+
# capabability.</em> {[RFC9394]}[https://rfc-editor.org/rfc/rfc9394]
#
# For example:
#
# # Without partial, the size of the results may be unknown beforehand:
# results = imap.uid_fetch(next_uid_to_fetch.., %w(UID FLAGS))
# # ... maybe wait for a long time ... and allocate a lot of memory ...
# results.size # => 0..2**32-1
# process results # may also take a long time and use a lot of memory...
#
# # Using partial, the results may be paginated:
# loop do
# results = imap.uid_fetch(next_uid_to_fetch.., %w(UID FLAGS),
# partial: 1..500)
# # fetch should return quickly and allocate little memory
# results.size # => 0..500
# break if results.empty?
# next_uid_to_fetch = results.last.uid + 1
# process results
# end
#
# >>>
# *Note:* Servers _MUST_ implicitly include the +UID+ message data item as
# part of any +FETCH+ response caused by a +UID+ command, regardless of
Expand All @@ -2424,9 +2476,10 @@ def fetch(set, attr, mod = nil, changedsince: nil)
# Related: #fetch, FetchData
#
# ==== Capabilities
#
# Same as #fetch.
def uid_fetch(set, attr, mod = nil, changedsince: nil)
fetch_internal("UID FETCH", set, attr, mod, changedsince: changedsince)
def uid_fetch(...)
fetch_internal("UID FETCH", ...)
end

# :call-seq:
Expand Down Expand Up @@ -3382,7 +3435,26 @@ def search_internal(cmd, ...)
end
end

def fetch_internal(cmd, set, attr, mod = nil, changedsince: nil)
def partial_range(range)
case range
in /\a(?:\d+:\d+|-\d+:-\d+)\z/
range
in Range
minmax = range.minmax.map { Integer _1 }
if minmax.all?(1..2**32-1) || minmax.all?(-2**32..-1)
minmax.join(":")
else
raise ArgumentError, "invalid partial-range"
end
end
end

def fetch_internal(cmd, set, attr, mod = nil, partial: nil, changedsince: nil)
set = SequenceSet[set]
if partial
mod ||= []
mod << "PARTIAL" << partial_range(partial)
end
if changedsince
mod ||= []
mod << "CHANGEDSINCE" << Integer(changedsince)
Expand All @@ -3399,9 +3471,9 @@ def fetch_internal(cmd, set, attr, mod = nil, changedsince: nil)
synchronize do
clear_responses("FETCH")
if mod
send_command(cmd, SequenceSet.new(set), attr, mod)
send_command(cmd, set, attr, mod)
else
send_command(cmd, SequenceSet.new(set), attr)
send_command(cmd, set, attr)
end
clear_responses("FETCH")
end
Expand Down
48 changes: 44 additions & 4 deletions lib/net/imap/esearch_result.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,16 +35,16 @@ def initialize(tag: nil, uid: nil, data: nil)

# :call-seq: to_a -> Array of integers
#
# When #all contains a SequenceSet of message sequence
# When either #all or #partial contains a SequenceSet of message sequence
# numbers or UIDs, +to_a+ returns that set as an array of integers.
#
# When #all is +nil+, either because the server
# returned no results or because +ALL+ was not included in
# When both #all and #partial are +nil+, either because the server
# returned no results or because +ALL+ and +PARTIAL+ were not included in
# the IMAP#search +RETURN+ options, #to_a returns an empty array.
#
# Note that SearchResult also implements +to_a+, so it can be used without
# checking if the server returned +SEARCH+ or +ESEARCH+ data.
def to_a; all&.numbers || [] end
def to_a; all&.numbers || partial&.to_a || [] end

##
# attr_reader: tag
Expand Down Expand Up @@ -135,6 +135,46 @@ def count; data.assoc("COUNT")&.last end
# and +ESEARCH+ {[RFC4731]}[https://www.rfc-editor.org/rfc/rfc4731.html#section-3.2].
def modseq; data.assoc("MODSEQ")&.last end

# Returned by ESearchResult#partial.
#
# Requires +PARTIAL+ {[RFC9394]}[https://www.rfc-editor.org/rfc/rfc9394.html]
# or <tt>CONTEXT=SEARCH</tt>/<tt>CONTEXT=SORT</tt>
# {[RFC5267]}[https://www.rfc-editor.org/rfc/rfc5267.html]
#
# See also: #to_a
class PartialResult < Data.define(:range, :results)
def initialize(range:, results:)
range => Range
results => SequenceSet | nil
super
end

##
# method: range
# :call-seq: range -> range

##
# method: results
# :call-seq: results -> sequence set or nil

# Converts #results to an array of integers.
#
# See also: ESearchResult#to_a.
def to_a; results&.numbers || [] end
end

# :call-seq: partial -> PartialResult or nil
#
# A PartialResult containing a subset of the message sequence numbers or
# UIDs that satisfy the SEARCH criteria.
#
# Requires +PARTIAL+ {[RFC9394]}[https://www.rfc-editor.org/rfc/rfc9394.html]
# or <tt>CONTEXT=SEARCH</tt>/<tt>CONTEXT=SORT</tt>
# {[RFC5267]}[https://www.rfc-editor.org/rfc/rfc5267.html]
#
# See also: #to_a
def partial; data.assoc("PARTIAL")&.last end

end
end
end
47 changes: 47 additions & 0 deletions lib/net/imap/response_parser.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -321,6 +321,20 @@ module RFC3629
SEQUENCE_SET = /#{SEQUENCE_SET_ITEM}(?:,#{SEQUENCE_SET_ITEM})*/n
SEQUENCE_SET_STR = /\A#{SEQUENCE_SET}\z/n

# partial-range-first = nz-number ":" nz-number
# ;; Request to search from oldest (lowest UIDs) to
# ;; more recent messages.
# ;; A range 500:400 is the same as 400:500.
# ;; This is similar to <seq-range> from [RFC3501]
# ;; but cannot contain "*".
PARTIAL_RANGE_FIRST = /\A(#{NZ_NUMBER}):(#{NZ_NUMBER})\z/n

# partial-range-last = MINUS nz-number ":" MINUS nz-number
# ;; Request to search from newest (highest UIDs) to
# ;; oldest messages.
# ;; A range -500:-400 is the same as -400:-500.
PARTIAL_RANGE_LAST = /\A(-#{NZ_NUMBER}):(-#{NZ_NUMBER})\z/n

# RFC3501:
# literal = "{" number "}" CRLF *CHAR8
# ; Number represents the number of CHAR8s
Expand Down Expand Up @@ -1517,6 +1531,9 @@ def esearch_response
# From RFC4731 (ESEARCH):
# search-return-data =/ "MODSEQ" SP mod-sequence-value
#
# From RFC9394 (PARTIAL):
# search-return-data =/ ret-data-partial
#
def search_return_data
label = search_modifier_name; SP!
value =
Expand All @@ -1526,11 +1543,41 @@ def search_return_data
when "ALL" then sequence_set
when "COUNT" then number
when "MODSEQ" then mod_sequence_value # RFC7162: CONDSTORE
when "PARTIAL" then ret_data_partial__value # RFC9394: PARTIAL
else search_return_value
end
[label, value]
end

# From RFC5267 (CONTEXT=SEARCH, CONTEXT=SORT) and RFC9394 (PARTIAL):
# ret-data-partial = "PARTIAL"
# SP "(" partial-range SP partial-results ")"
def ret_data_partial__value
lpar
range = partial_range; SP!
results = partial_results
rpar
ESearchResult::PartialResult.new(range, results)
end

# partial-range = partial-range-first / partial-range-last
# tagged-ext-simple =/ partial-range-last
def partial_range
case (str = atom)
when Patterns::PARTIAL_RANGE_FIRST, Patterns::PARTIAL_RANGE_LAST
min, max = [Integer($1), Integer($2)].minmax
min..max
else
parse_error("unexpected atom %p, expected partial-range", str)
end
end

# partial-results = sequence-set / "NIL"
# ;; <sequence-set> from [RFC3501].
# ;; NIL indicates that no results correspond to
# ;; the requested range.
def partial_results; NIL? ? nil : sequence_set end

# search-modifier-name = tagged-ext-label
alias search_modifier_name tagged_ext_label

Expand Down
1 change: 1 addition & 0 deletions rakelib/rfcs.rake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,7 @@ RFCS = {
8514 => "IMAP SAVEDATE",
8970 => "IMAP PREVIEW",
9208 => "IMAP QUOTA, QUOTA=, QUOTASET",
9394 => "IMAP PARTIAL",

# etc...
3629 => "UTF8",
Expand Down
Loading
Loading