Slow and broken paste, if too long.

[estani]

Description

Pasting is still slow as described in #115 and #43
Pasting this long line fails:

numbers = [484, 589, 633, 634, 746, 749, 825, 826, 855, 863, 864, 891, 911, 913, 961, 975, 993, 998, 1036, 1045, 1063, 1064, 1065, 1066, 1067, 1069, 1088, 1089, 1107, 1108, 1110, 1111, 1124, 1128, 1129, 1130, 1136, 1141, 1155, 1160, 1161, 1181, 1186, 1187, 1190, 1206, 1212, 1213, 1218, 1235, 1240, 1242, 1243, 1251, 1252, 1261, 1262, 1268, 1269, 1276, 1278, 1281, 1285, 1286, 1287, 1295, 1299, 1300, 1302, 1304, 1305, 1307, 1312, 1314, 1316, 1317, 1318, 1321, 1333, 1336, 1338, 1347, 1349, 1352, 1354, 1355, 1356, 1358, 1361, 1364, 1365, 1366, 1375, 1377, 1388, 1391, 1393, 1400, 1403, 1406, 1407, 1411, 1412, 1418, 1419, 1421, 1427, 1429, 1432, 1437, 1439, 1442, 1447, 1448, 1449, 1455, 1456, 1457, 1458, 1459, 1460, 1469, 1473, 1474, 1475, 1476, 1485, 1486, 1490, 1497, 1500, 1503, 1504, 1507, 1519, 1520, 1521, 1525, 1526, 1531, 1534, 1539, 1540, 1541, 1542, 1545, 1551, 1552, 1554, 1557, 1561, 1562, 1565, 1566, 1569, 1570, 1575, 1577, 1580, 1581, 1582, 1583, 1584, 1585, 1590, 1593, 1595, 1598, 1608, 1609, 1611, 1612, 1614, 1615, 1616, 1623, 1629, 1637, 1638, 1640, 1641, 1646, 1649, 1651, 1659, 1660, 1662, 1663, 1664, 1666, 1667, 1673, 1675, 1676, 1678, 1681, 1684, 1685, 1686, 1692, 1699, 1700, 1705, 1706, 1715, 1718, 1719, 1724, 1735, 1736, 1737, 1740, 1743, 1744, 1752, 1753, 1754, 1755, 1756, 1757, 1761, 1765, 1772, 1774, 1776, 1780, 1783, 1786, 1787, 1789, 1790, 1793, 1794, 1798, 1801, 1803, 1804, 1809, 1813, 1814, 1821, 1822, 1824, 1832, 1833, 1834, 1835, 1838, 1839, 1842, 1843, 1846, 1850, 1865, 1866, 1867, 1868, 1870, 1873, 1874, 1877, 1879, 1892, 1894, 1898, 1899, 1900, 1902, 1903, 1904, 1906, 1907, 1908, 1910, 1914, 1915, 1916, 1918, 1932, 1935, 1937, 1943, 1946, 1947, 1948, 1953, 1956, 1957, 1958, 1959, 1964, 1968, 1969, 1972, 1973, 1981, 1982, 1990, 1993, 1994, 1995, 1996, 1999, 2000, 2020, 2021, 2022, 2023, 2024, 2029, 2030, 2032, 2033, 2034, 2036, 2037, 2045, 2049, 2052, 2053, 2054, 2055, 2056, 2057, 2058, 2063, 2065, 2066, 2068, 2071, 2081, 2082, 2083, 2084, 2085, 2087, 2088, 2093, 2094, 2095, 2097, 2099, 2102, 2106, 2107, 2108, 2109, 2113, 2114, 2115, 2116, 2117, 2119, 2120, 2121, 2124, 2126, 2127, 2128, 2129, 2130, 2134, 2135, 2136, 2137, 2138, 2139, 2140, 2141, 2142, 2144, 2145, 2146, 2147, 2148, 2150, 2155, 2156, 2158, 2160, 2161, 2163, 2164, 2170, 2176, 2177, 2179, 2180, 2182, 2184, 2188, 2189, 2194, 2198, 2199, 2202, 2204, 2205, 2212, 2213, 2214, 2215, 2218, 2219, 2220, 2223, 2225, 2233, 2234, 2241, 2243, 2244, 2245, 2246, 2247, 2248, 2249, 2254, 2255, 2256, 2259, 2260, 2264, 2265, 2266, 2267, 2268, 2269, 2271, 2276, 2277, 2278, 2279, 2286, 2287, 2288, 2289, 2293, 2294, 2295, 2297, 2298, 2303, 2304, 2307, 2309, 2310, 2311, 2312, 2313, 2317, 2318, 2319, 2320, 2321, 2322, 2323, 2325, 2326, 2327, 2328, 2330, 2332, 2334, 2337, 2338, 2340, 2343, 2344, 2345, 2346, 2347, 2348, 2352, 2353, 2355, 2357, 2358, 2359, 2363, 2365, 2366, 2372, 2374, 2375, 2379, 2380, 2381, 2383, 2384, 2385, 2388, 2389, 2390, 2391, 2392, 2393, 2395, 2397, 2398, 2399, 2405, 2406, 2407, 2408, 2409, 2410, 2411, 2413, 2415, 2416, 2417, 2420, 2422, 2423, 2438, 2440, 2441, 2445, 2446, 2447, 2448, 2449, 2450, 2451, 2454, 2455, 2457, 2458, 2459, 2460, 2461, 2465, 2466, 2469, 2470, 2471, 2472, 2477, 2478, 2480, 2481, 2482, 2483, 2484, 2485, 2486, 2491, 2494, 2496, 2497, 2498, 2499, 2500, 2502, 2503, 2504, 2505, 2507, 2508, 2509, 2517, 2518, 2519, 2520, 2521, 2522, 2523, 2524, 2526, 2527, 2528, 2529, 2530, 2531, 2532, 2533, 2534, 2541, 2542, 2543, 2544, 2545, 2546, 2547, 2548, 2549, 2552, 2558, 2559, 2560, 2561, 2562, 2563, 2564, 2565, 2566, 2567, 2568, 2575, 2576, 2577, 2581, 2582, 2583, 2584, 2585, 2587, 2588, 2589, 2594, 2595, 2597, 2598, 2600, 2603, 2604]

One part (up to the end) is immediately pasted, the other takes a long while, and stops there, leaving a half pasted line.
When starting irb with --nomultiline this does not happens, and everything works perfectly.

Stoping irb and/or starting a new one with this option is a workaround, but sometimes very hard to do, as state need to be recreated.

Not sure this is fixable, but vim already have a good solution for these kind of problems, and that's to set a "paste mode".

Proposal

To have a flag that activates/deactivates the multiline support from within irb. Is that even possible?
Normally the coloring and multiline support works quite good (besides the line breaks added that make copying text from irb also a pain), just now and then it would be great to turn the feature temporarily off...

Terminal Emulator

term2

[aycabta]

Would you paste the result of irb_info command in IRB?

[ayosec]

I have the same issue.

Would you paste the result of irb_info command in IRB?

>> irb_info
=>
Ruby version: 2.7.4
IRB version: irb 1.2.6 (2020-09-14)
InputMethod: ReidlineInputMethod with Reline 0.1.5
.irbrc path: /home/ayosec/.config/ruby/irb-conf.rb

In ~/.config/ruby/irb-conf.rb:

require "irb/completion"

IRB.conf[:PROMPT_MODE] = :SIMPLE
IRB.conf[:AUTO_INDENT] = true

Example

Pasting an array with 200 numbers:

[ayosec]

If I add IRB.conf[:USE_MULTILINE] = false to the irb-conf.rb file it is fast again (but with no syntax highlight).

[aycabta]

@ayosec Could you try the latest version of irb gem and reline gem by gem install irb reline? The reline gem handles the speedup process, and you are using a too old reline gem.

[ayosec]

@aycabta With the latest versions of irb and reline it is much faster, but still quiet slower than using IRB.conf[:USE_MULTILINE] = false.

I noticed that with the default input method, it tries to redraw the prompt line for each character. I captured the calls to the write syscall and it shows 3177 calls:

$ strace -r -e write -s1024 -o /tmp/writes irb
paste a big array in irb from the clipboard

$ wc -l /tmp/writes
3177 /tmp/writes

I uploaded the file to https://gist.github.com/ayosec/d1427cbf75a26ede86718b60545743a2 in case it can be useful.

By contrast, with :USE_MULTILINE = false it only does 44 calls to write.

[aycabta]

@ayosec I know that the price of multiple features is a lot of extra function calls. Are there any situations where it is dysfunctional in practical use?

[ayosec]

For me, the practical use is just disable multiline, so I'm fine with the current status. I can't speak for other users, tho.

A possible solution that may work always is to support bracketed paste mode in irb. This is enabled by default in most programs I use daily (like vim or bash). Using this feature, available in almost every terminal emulator nowadays, irb does not need to redraw the prompt after every character; instead, it waits until CSI 2 0 1 ~ is received, and then it updates the editor.

[estani]

I think there are a lot of options to improve this:

• allow from within the IRB to turn paste mode on/off (best and simpler work around)
• redraw only on certain characters, not all (e.g. brackets and/or '.', '=', etc) see @ayosec previews comment
• redraw with a delay (e.g. 500ms), every keystroke resets the timer (this should fix it for every case).

Would be great if any of these makes it into the code (whichever simpler is).

[aycabta]

A possible solution that may work always is to support bracketed paste mode in irb.

• allow from within the IRB to turn paste mode on/off (best and simpler work around)
• redraw only on certain characters, not all (e.g. brackets and/or '.', '=', etc) see @ayosec previews comment
• redraw with a delay (e.g. 500ms), every keystroke resets the timer (this should fix it for every case).

Most of them have already been introduced or rejected as a result of actual implementation and testing, and various other speedup measures have already been introduced. Interactive environments on virtual terminals are extremely complex, and it is impossible to combine rich UI, execution speed, and development speed. The other reasonably well-known programming language that bundles a "standard" and "very rich" REPL like IRB is only Julia. This is because it requires a constant effort to make the impossible possible, and it has to support a really large number of users. People don't want to do it because it's not realistic.

Thanks for all the reports and feedbacks. I'll try to think of more ways to improve it when I have more time, but please understand that I spend tens or hundreds of hours each time just to make one small speedup.

[estani]

The possibility to turn Multiline on and off from within the cli would be enough, even if the history is lost (in case that's possible)

[alfuken]

The possibility to turn Multiline on and off from within the cli would be enough, even if the history is lost (in case that's possible)

I second that. Having a configuration option is nice, but it can't be applied to every IRB instance, not, for example, to a Rails console on a production server or someone else's project.
No matter how good you optimise it, it will still be slower than if used with IRB.conf[:USE_MULTILINE] = false.
Features are nice, but not when they impact speed and introduce lag. People somehow managed to survive without coloured highlighting and autocomplete for years, I think most of us can still manage to live without them in future as well.

It would be really helpful to have an option to switch between fancy and fast from the console directly, even if that means losing history and/or variables. Essentially reloading IRB in a non-multiline mode directly from within IRB session would also do.

[jrochkind]

For me it's not just that it's slow, it's that it gets garbled.

Trying to paste 1900 bytes of a ruby string literal of serialized JSON into IRB, it winds up out of order and missing some parts, it's corrupted I can't actually use it to paste some data into IRB.

(This isn't reproducible with arbitrary 1900 bytes of ascii, it's happening with a string that has lots of \" in it, I guess triggering parsing of some kind. Like I'm trying to paste something like this into IRB: "{\"id\":5572,\"name\":\"Smith, Kohn\",...". except 1900 characters of that. It winds up corrupt/garbled/missing some parts)

If I set IRB.conf[:USE_MULTILINE] = false in .irbrc, I can paste in my 1900 byte string with no corruption.

The ability to toggle this from within an IRB session would at least let me do so before trying to paste the thing.

[tompng]

Broken paste is fixed in Reline 0.3.3 (maybe ruby/reline#474)
Paste speed is improved in Reline 0.5.0 (improve rendering) and in Reline 0.5.5 (bracketed paste ruby/reline#655)