Stop Opening Huge Files in Screen Editors

[r1w1s1]

I came across this post:

https://phanpy.social/#/hachyderm.io/s/115891592999188880

It describes a very common situation: large SQL dumps, vim becoming
unusable, and ed(1) suddenly making sense again.

This is not accidental.

Screen editors are designed for interactive, cursor-based editing.
When files reach hundreds of megabytes or gigabytes, those assumptions break down.
Startup cost, redraw, navigation, and internal structures become the real bottleneck.

At that point, using tools like grep, sed, awk, or ed is not a
workaround — it is the correct approach.

Key idea:

Large files do not require faster editors;
they require different tools.

Full note:
https://codeberg.org/r1w1s1/code-notes/raw/branch/main/notes/Stop_Opening_Huge_Files_in_Screen_Editors.txt

[naltun]

Great reminder. I used vim to dig through a massive SQL dump during a CTF and although I performed well it wasn't without some headache.

[TomAoki]

Maybe not so useful for many, but some screen editors are not good at handling looooong single line, even if the whole bunch of file size is small.

An example of this situation would be copy (from terminal emulator) and paste to editor window the output from poudriere to see specific ports are included in skipped (or succeeded) ones on building a large number of ports. Built, ignored, failed, skipped ports has single output line respectively.

In this specific cases, editors/leafpad (and IIRC, editors/pluma, too) locks up on horizontal scrolling (in case turned over at right end of window, vertical, too) or searching. devel/geany works fine.
Just examples I've tried before.

[mhd]

As another data point in comparing how different editors work, here's an old one by the author of the very undervalued joe editor.

In general, there's nothing preventing screen editors from working properly and fast here, it's just not something they optimized for. If they did it at all, internal editor data structures tend to be quite simple these days, as virtual memory works for most cases. People had to get a lot more tricky on smaller devices (or they gave up early, cf. Windows Notepad).

Heretically, I'd also like to say that a lot of files that aren't binary and huge, different tools often wouldn't point at e.g. sed, as you're running into the old "Now they have two problems." issue then.