Opening files like docx requires heavy string parsing and pointer allocation. After an OS swaps active RAM pages out as to flash disk as virtual memory, it can restore them almost instantly. Using 2X more flash disk space is worth if we can achieve millisecond loads. Why hasn't direct memory dumping/reloading replaced file parsing for complex document models?
I started a company in 2000 to build a commercial cross-platform desktop suite in Java. To achieve seamless live-data-linking across text, sheets and slides, we stored data objects into a 3D coordinate space (Sheet_Num,Row_Num,Column_Num) and reference/access them instead of pointers. This scheme enable us open a 50,000-pages document in 8 seconds, compared to 300+ seconds by a competing suite requiring parsing andan array of pointer construction.
Recently, some suggested that our 3D design without referencing data by pointers can possibly use OS provided mmap to restore back the entire document image as loaded into the RAM to flash disk. Instead of opening the file again with all those parsing/pointer-building work, the memory mapped buffers can be simply be brought back into RAM in milliseconds.
Our senior engineers have confirmed the feasibility and tries to prove it, but, they have encountered numerous problems to save using OpenJDK and CRaC on Linux. They told me that OpenSDK has stated that it can be done but probably did not test this part thoroughly because they never expected it be actually used.
Has any one successfully decouple a complex document substrate into off-heap memory to achieve true zero-copy mmap load or does the JVM runtime always get in the way?
So we can share them with each other? Are we going to pass around memory dumps instead? What about tool choice when working on a specific document format?
- Memory might not look the same on all platforms, if your app is multi-platform you stop being able to share your data cross-platform.
- The internal data structures of your app will absolutely change as your app evolves, the binary data in memory is a raw result of your data structures. So you have to commit to never changing data structures, or create complicated binary migration tools to update memory when your app updates.
- Debugging corrupted memory is incredibly tedious any sometimes impossible. Having plain text serialized data makes debugging much more straightforward.
Top my my head I would suggest to focus more on improving performance on your serialization pipeline. There might be some subset of your data that is unlikely to ever change and is identical on all platforms, maybe your serializer is hybrid in that case.
When you use RandomAccessFile API in JVM one of the flags allows you to mmap the contents
try (RandomAccessFile file = new RandomAccessFile("example.dat", "rw");
FileChannel channel = file.getChannel()) {
// Map the file into memory from position 0 up to bufferSize
MappedByteBuffer buffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, bufferSize);
Once you have a mappedbytebuffer you can use MemoryLayout from FFM to view it as structured data without deserializing
I hear the old Microsoft Word save files used memory dumps, which became an undocumented nightmare of a file format.
You need total control over memory layout of the saved runtime state. No hidden fields. No pointers either - you want to be able to reload at another address. Sooner or later, you might want to add some state, and you'd need a place to put it.
That said, I have used mmap something like this, mostly so state (data) could be paged in on demand.
I started a company in 2000 to build a commercial cross-platform desktop suite in Java. To achieve seamless live-data-linking across text, sheets and slides, we stored data objects into a 3D coordinate space (Sheet_Num,Row_Num,Column_Num) and reference/access them instead of pointers. This scheme enable us open a 50,000-pages document in 8 seconds, compared to 300+ seconds by a competing suite requiring parsing andan array of pointer construction.
Recently, some suggested that our 3D design without referencing data by pointers can possibly use OS provided mmap to restore back the entire document image as loaded into the RAM to flash disk. Instead of opening the file again with all those parsing/pointer-building work, the memory mapped buffers can be simply be brought back into RAM in milliseconds.
Our senior engineers have confirmed the feasibility and tries to prove it, but, they have encountered numerous problems to save using OpenJDK and CRaC on Linux. They told me that OpenSDK has stated that it can be done but probably did not test this part thoroughly because they never expected it be actually used.
Has any one successfully decouple a complex document substrate into off-heap memory to achieve true zero-copy mmap load or does the JVM runtime always get in the way?
So we can share them with each other? Are we going to pass around memory dumps instead? What about tool choice when working on a specific document format?
Serializing has many uses.
- Memory might not look the same on all platforms, if your app is multi-platform you stop being able to share your data cross-platform.
- The internal data structures of your app will absolutely change as your app evolves, the binary data in memory is a raw result of your data structures. So you have to commit to never changing data structures, or create complicated binary migration tools to update memory when your app updates.
- Debugging corrupted memory is incredibly tedious any sometimes impossible. Having plain text serialized data makes debugging much more straightforward.
Top my my head I would suggest to focus more on improving performance on your serialization pipeline. There might be some subset of your data that is unlikely to ever change and is identical on all platforms, maybe your serializer is hybrid in that case.
When you use RandomAccessFile API in JVM one of the flags allows you to mmap the contents
Once you have a mappedbytebuffer you can use MemoryLayout from FFM to view it as structured data without deserializing
Be careful!
I hear the old Microsoft Word save files used memory dumps, which became an undocumented nightmare of a file format.
You need total control over memory layout of the saved runtime state. No hidden fields. No pointers either - you want to be able to reload at another address. Sooner or later, you might want to add some state, and you'd need a place to put it.
That said, I have used mmap something like this, mostly so state (data) could be paged in on demand.