Skip to content

Loading a large file into an editor

A 3GB AmigaDOS disk dump does not load into TextWrangler

So Ingo tried to load a 3GB file into TextWrangler on MacOS, and that did not work despite him having plenty of memory left. That’s of course, because TextWrangler is still a 32 Bit binary:

$ pwd
/Applications/TextWrangler.app/Contents/MacOS
$ file * /bin/ls
TextWrangler: Mach-O executable i386
/bin/ls:      Mach-O 64-bit executable x86_64

and even with a theoretical limit of 4GB in a 32 Bit application memory space, a 3GB allocation won’t fit. In MacOS, you can use vmmap to look at the contents of a processes memory map:

$ ps auxwww | grep Text[W]rangler
kris               349   0.0  0.1   848392   9468   ??  S     5Aug17   0:56.75 /Applications/TextWrangler.app/Contents/MacOS/TextWrangler -psn_0_73746
$ vmmap 349
Process:         TextWrangler [349]
Path:            /Applications/TextWrangler.app/Contents/MacOS/TextWrangler
Load Address:    0xdd000
Identifier:      com.barebones.textwrangler
Version:         5.5.2 (397016)
Code Type:       X86
Parent Process:  ??? [1]
 
Date/Time:       2017-08-27 14:07:28.908 +0200
Launch Time:     2017-08-05 16:08:34.755 +0200
OS Version:      Mac OS X 10.12.3 (16D32)
Report Version:  7
Analysis Tool:   /Applications/Xcode.app/Contents/Developer/usr/bin/vmmap32
Analysis Tool Version:  Xcode 8.3.3 (8E3004b)
----
 
Virtual Memory Map of process 349 (TextWrangler)
Output report format:  2.4  -- 32-bit process
VM page size:  4096 bytes
 
==== Non-writable regions for process 349
REGION TYPE              START - END     [ VSIZE  RSDNT  DIRTY   SWAP] PRT/MAX SHRMOD PURGE    REGION DETAIL
__TEXT                 000dd000-0066c000 [ 5692K  2200K     4K     8K] r-x/rwx SM=COW          ...xtWrangler.app/Contents/MacOS/TextWrangler
...
__TEXT                 9e22c000-9e248000 [  112K     4K     0K     0K] r-x/r-x SM=COW          /usr/lib/libCRFSuite.dylib
__TEXT                 9e248000-9e252000 [   40K     4K     0K     0K] r-x/r-x SM=COW          /usr/lib/libChineseTokenizer.dylib
__TEXT                 9e252000-9e254000 [    8K     8K     0K     0K] r-x/r-x SM=COW          /usr/lib/libDiagnosticMessagesClient.dylib
...
__TEXT                 9f9d4000-9f9fb000 [  156K   140K     0K     0K] r-x/r-x SM=COW          /usr/lib/system/libxpc.dylib
__IMAGE                a9419000-a949d000 [  528K     4K     0K     0K] r--/r-- SM=COW          ...meworks/AppKit.framework/Versions/C/AppKit
__UNICODE              a949d000-a9528000 [  556K   316K     0K     0K] r--/r-- SM=COW          ...dation.framework/Versions/A/CoreFoundation
__GLSLBUILTINS         a9528000-a97af000 [ 2588K   348K     0K     0K] r--/r-- SM=COW          ...ons/A/Libraries/libGLProgrammability.dylib
__LINKEDIT             a97af000-ac9b4000 [ 50.0M  5388K     0K     0K] r--/r-- SM=COW          dyld shared cache combined __LINKEDIT
STACK GUARD            b01a7000-b01a8000 [    4K     0K     0K     0K] ---/rwx SM=NUL          stack guard for thread 1
STACK GUARD            bbf24000-bf724000 [ 56.0M     0K     0K     0K] ---/rwx SM=NUL          stack guard for thread 0
shared memory          ffff0000-ffff1000 [    4K     4K     4K     0K] r--/r-- SM=SHM
shared memory          ffffe000-fffff000 [    4K     4K     4K     0K] r-x/r-x SM=SHM

You can see that kernel memory is being allocated to the upper 1GB (starting at 0xc0*), and the lower 3 GB have to accommodate a per-thread stack (in 0xb*), and all shared, dynamically loaded code, which happens to be not very densely packed in 0x9* and 0xa* due to ASLR. That leaves roughly 2GB for data.

Emacs won’t load a 3GB file, either, because it does unspeakable things to address pointer bits, and even in a 64-bit build does have limitations of some 256 MB or comparable by default.

vim works just fine, takes about one minute to handle this.

 
$ time dd if=/dev/random of=keks2 bs=1024k count=3072 
3221225472 bytes transferred in 251.311621 secs (12817654 bytes/sec)
$ time /usr/local/vim keks2 # homebrew
...
real	1m01.322s
...
Published inApple

10 Comments

  1. kris kris

    Just for the record, joe works and takes about 1m10s. nano takes about 4m, but also manages to load the file.

  2. JC

    64 Bit gibt’s hier:

    /usr/bin/emacs: Mach-O 64-bit executable x86_64
    Aquamacs: Mach-O 64-bit executable x86_64

    Der schon etwa ältere Emacs 22.1.1 den ich gerade unter OSX 10.9.5 zugreifbar habe, lädt ein File mit 1.1G und auch eines mit 1.6G problemlos. Und das in unter einer Minute.

    Ob man GB-Files wirklich im Editor und nicht besser mit einem anderen Werkzeug bearbeitet, sei mal ignoriert.

  3. Bernd Wachter

    Where exactly did you get that emacs information from? It’s not on that page you’ve linked, and at least on Linux emacs 64bit happily opens large files.

  4. Rudolf Polzer

    Boring. Everyone who worked on a game engine of this millennium had to implement workarounds for the tiny address space to make the game still work on Windows XP.

  5. Jürgen Lanzki

    Ich nutze BBEdit, habe aber noch nie so große Dateien damit bearbeitet.

  6. Timo Buhmann

    That may be the reason why barebone is working on a 64 bit version of BBEdit and Textwrangler is already end of life anyways.

Leave a Reply

Your email address will not be published. Required fields are marked *