ArchiveFile:ReadAll Mangles files by encoding and decoding to utf8 when undesired

[BAM5]

I thought I was being clever, making 128 different files with a single byte value in it to give me the ability to write a byte of any value to a file...
My god what a headache. I spent hours into the night figuring out what the issue was. It was this one line of code...

KOS/src/kOS.Safe/Persistence/ArchiveFile.cs

Line 23 in 8f281a4

bytes = Archive.ConvertFromWindowsNewlines(bytes);

You see, this function decodes and encodes any data, binary or not, if it doesn't pass some coarse check to see if it ISN'T a string. In my case, this function will categorize the file as "TOO SHORT" and this function throws up its hands and says "Screw it! It's a string!" This has ruined me royally.

I request a bit better logic here. This whole issue is revolving around this function that swaps windows linebreak sequence with the typical singular newline character. So the simplest thing to do is leave practically all the logic as is, and then at the end, just do a length check of the original decoded string and the string after replacement. If it's shorter, I suppose we can assume that it was a string and encode and return it. If not, just return the original bytes.

I guess this won't work for binary files with strings embedded though so...
Maybe we move the function to the FileContent:String function? Then it will only execute when the consumer wants the data as a string. When they request the Binary, they'll get the original data.

Or maybe just have a check in the ArchiveFile:ReadAll method that looks at the file extension and skips ConvertFromWindowsNewlines if it's ".bin" or something. At least this way there's a clear flag from user-space that gives us the intention of the file.

It's 3:20. I'm going to sleep. Upset.

[JonnyOThan]

It looks like KOS already has some support for dealing with binary files. The key issue, as you note, is that it tries to auto-detect the file format and needs at least 4 bytes.

I think it may be reasonable to make this check smarter, and have it return BINARY in certain cases even if there are fewer than 4 bytes. Specifically, when it contains any non-ascii characters. However that likely won't fix your use case when the file contains a newline character but you want to treat it as binary.

Perhaps a better solution would be to provide a way to open a file in binary mode.

[BAM5]

However that likely won't fix your use case when the file contains a newline character but you want to treat it as binary.

Luckily \n is within the 0-127 range that file:Write(CHAR(n)) can handle. I ran into this problem because this was my workaround for writing byte values above 127 to a file using the VolumeFile:Write(data:FileContent) overload function which allows me to write the raw binary (or so I thought) of one file to another without invoking that pesky utf8 encoding.

Perhaps a better solution would be to provide a way to open a file in binary mode.

That'd be great! But seems like significantly more investment as it'd be a new feature.

[BAM5]

Not that it changes anything, but I did a work around for the files... So that I can do my work around for writing bytes to files.
This issue only affects files loaded through the archive, so I proompt'd gemini to make a script that will create a list of files I can inject into the save file directly to bypass ArchiveFile:Readall.

After a few iterations here is the result

async function generateKOSSaveString() {
  // Create an array of numbers from 128 to 255
  const fileIndices = Array.from({ length: 128 }, (_, i) => i + 128);

  // Map each index to a Promise that resolves to its formatted file string
  const filePromises = fileIndices.map(async (i) => {
    // 1. Create the 1-byte content
    const fileContent = new Uint8Array([i]);

    // 2. Gzip compress in parallel
    // Note: We create a new stream for each file
    const stream = new Blob([fileContent]).stream().pipeThrough(new CompressionStream('gzip'));
    const response = new Response(stream);
    const compressedBuffer = await response.arrayBuffer();

    // 3. Convert to Base64 using native method
    const bytes = new Uint8Array(compressedBuffer);
    let base64 = bytes.toBase64();

    // 4. Replace all '/' with ',' (using replaceAll for clarity)
    // This is important since kos does the inverse operation before base64 decoding. Something about having two "/" in a row being interpreted as a comment.
    base64 = base64.replaceAll('/', ',');

    // 5. Format the individual file entry
    return `file {\n\tfilename = ${i}\n\tbinary = ${base64}\n}`;
  });

  // Wait for all compression tasks to finish
  const files = await Promise.all(filePromises);

  // Join them all together
  return files.join('\n');
}

// Execute and log/copy
generateKOSSaveString().then(result => {
    // Encapsulated in an object for ease of copying property string from console
    console.dir({result});
});

And the result!

[BAM5]

Oh man, it even does it when writing binary files from the vessel's volume TO the archive volume. RIP.
I guess I just have to do everything on the local volume. So many work arounds.

[JonnyOThan]

That'd be great! But seems like significantly more investment as it'd be a new feature.

But it’s not really a new feature. Binary files are already supported, they just have to be more than 4 bytes and not appear to contain a string. All it would take is a new function to make it explicit.

[BAM5]

It could work, I'm not against it. It would just be very involved. All the file io functions use the readall method to get the file contents, which is where the mangling happens with ArchiveFile class. So if you wanted to have full support for binary files and give a consistent user experience you'd need to add a flag to the File IO functions like copy / move path and open because there's no way to intrinsically mark a file as binary unless through an explicit file extension like I suggested before. Without modifying them, the data is corrupted as its copied from the archive to the local volume, and so opening the file as binary after the fact doesn't help. Also, there's an issue with writing binary filecontent to an archive file as it undergoes the same string encoding/decoding process somewhere that mangles the data before it is written to the archive file.

This is why I was trying to suggest a simple fix. :)

If we're adding functions though, in my case, it would be so much easier if the the Write function could take an integer value between 0 and 255 and write it as a byte, this whole ordeal I've been working through is just to do that. Could also just be a separate function called WriteByte. It would also be extremely useful to have bitwise operators available. And hex value parsing in kerboscript.

As a side note, I've found what I believe is the best work around for now. I didn't realize the Write/Read JSON functions do full blown object serialization.
I was trying to make a simple lexicon builder that would operate in a single instruction doing basically READJSON(FileData["{128:0,129:0,130:0...etc}"]) but it was throwing errors. I figured out it was doing object serialization when I tried to use WriteJSON to see what was expected as the ReadJSON input.

When I realized this I looked up FileContent and saw it is serializable. So now I've created a monolithic JSON file that contains a serialized list of all instances of the byte files' FileContent, allowing me to load all the byte values into a list in one operation.
And now, since the file is utf8 and all the FileContent's binary data is base64 encoded, I no longer have issues with kos mangling the data! I can easily transfer this file from and to the archive with no corruption.
However... The file is HUGE, just over 17kB to store 128 bytes of binary data because of all the serialization overhead and the FileContent's gzip overhead & base64 encoding the data.
What's ironic though is doing it this way significantly reduces complexity and size in the KSP save file. It's like 1600 characters since it is stored as a single, gzipped, file entry and gzip can remove the repetition of all the serialization & base64 file content. Having individual byte files on the local volume results in 13.3k characters of data spread across 128 file entries in the save file.

If I could make a suggestion though... Maybe the serialization logic should remove repetitive type values and just put them in a value table like how the KSM format does. This alone would reduce the raw JSON size by like 4KB in my case.