File I/O in Kotlin Native

I worked out how to do basic string file input and output for Kotlin Native using their standard POSIX libraries. The code for these methods is at the bottom. This article explores it in more detail if you are interested.

Kotlin Native (KN) allows developers to use a lot of the same libraries and idioms that they use for building applications on the JVM but compiling down to native executables. However it is not just bundling up a JVM and making it feel like a native executable, it’s actually a native executable. That has the downside of losing all of your favorite JVM libraries that we often rely on. Pure Kotlin libraries can be made to compile easily for whichever platform you are targeting your KN binary for. There are a growing number of libraries for KN and Kotlin Multiplatform, which also allows targeting of iOS, Android, JavaScript, etc. One thing that is still missing however from the Kotlin standard libraries for KN is basic File I/O. That’s the bad news. The good news is that with a little C-know how you can easily write your own using the POSIX library bindings built into KN for all of the desktop deployment options including Windows.

When I first started playing with Kotlin Native (KN) I was very stoked about how much of Kotlin I was able to use. I was all set to try my hand at writing a simple file processing app that I would have normally written in C# or Kotlin proper. I already had an HTTP client example running so this seemed like an easy step. It was easy until I realized there was no equivalent to java.io and java.nio in the Kotlin standard library. You’ll see lots of kotlin.* libraries which have java.* equivalents. When doing JVM-based Kotlin you think to yourself, “What’s the point of that?” Kotlin deployment to non-JVM platforms is the point! While there were Kotlin library platforms for network communications, NoSQL databases, and the like simple file I/O was missing.

My file I/O needs are pretty pedestrian. I’m usually just parsing a text file that has tabular data or JSON data. In Java/Kotlin or C# it’s usually something as simple as File.readAllLines(filepath) or File.writeString(filepath, lines) but never more complex than opening up a file stream and manually doing the above in a few lines of code. Would KN leave me with no ability to do that easily? Absolutely not, especially if one has any experience doing file I/O in C. Because KN is meant for developing code similiar to what you’d use lower level languages like C/C++ and Rust for, it has Platform Libraries which expose those capabilities too. This provides access to OpenGL, zlib, and other standard libraries but most importantly POSIX. When we think POSIX many people think UNIX, but actually Windows has a POSIX layer as does macOS. So using these POSIX libraries we can get the file I/O we need. That’s only part of the story though.

Saying that one has access to low level libraries sounds nice in theory but how is it in practice? If I have a Kotlin String that I want to write to the disk do I have to go through a bunch of machinations to turn it into a native C char * or worse isolate the code is some weird bridge layer like I would under JNI? The answer thankfully is nope. The Kotlin team has made the process seamless. With a few code annotations and some helper functions it’s possible to file I/O almost as easily as under Java. Below are the entirety of the functions I need to replace the equivalent methods I’d use under Java/C#.

First let’s look at reading a text file into a Kotlin string.

fun readAllText(filePath: String): String {
    val returnBuffer = StringBuilder()
    val file = fopen(filePath, READ_MODE) ?: 
        throw IllegalArgumentException("Cannot open input file $filePath")

    try {
        memScoped {
            val readBufferLength = 64 * 1024
            val buffer = allocArray<ByteVar>(readBufferLength)
            var line = fgets(buffer, readBufferLength, file)?.toKString()
            while (line != null) {
                returnBuffer.append(line)
                line = fgets(buffer, readBufferLength, file)?.toKString()
            }
        }
    } finally {
        fclose(file)
    }

    return returnBuffer.toString()
}

For people familiar with Kotlin/Java the call signature is pretty much identical. With this code you pass in a Kotlin String object for the file path and get back a populated String with all the data in the file. It is identical to the Java File.readString​(Path path). The inside may look a little odd but it’s pretty straight forward, especially to C users. In C the function for getting a file handle is fopen and the function for reading data from that file is fgets. So it’s as simple as opening a file handle, then closing it when we are ultimately done with fclose, reading the data and shoving into the Kotlin String. We just need to help it out a bit since we are now calling into unmanaged C code.

First we define a block where we are doing this in a memScoped region. This provides a region where Kotlin will clean up our natively allocated memory for us when we go out of scope. That is memory that is allocated with the allocArray method where we built the buffer. C does none of the memory management for you so it our responsibility to allocate a buffer, tell the C function how big it is when it tries to fill it, and then deallocate it when we are finished. As we can see the code keeps asking for lines up to 64K in length until it runs out of data at which point it returns null. Since converting from the byte arrays to Kotlin strings is a common thing Kotlin provides us with the toKString extension method for byte arrays. Lastly we want to each line to our StringBuilder. That’s it! Simply dropping this code into your Kotlin project now gives you the ability to read text files in the way you are used to. How about writing them?

First let’s look at the simple case of writing one giant string buffer to a file. Ironically Java doesn’t have a convenient method for this like C# does with its File.WriteAllText method. We will reproduce the C# signature instead:

fun writeAllText(filePath:String, text:String) {
    val file = fopen(filePath, WRITE_MODE) ?: 
        throw IllegalArgumentException("Cannot open output file $filePath")
    try {
        memScoped {
            if(fputs(text, file) == EOF) throw Error("File write error")
        }
    } finally {
        fclose(file)
    }
}

This is much simpler than reading isn’t it? Once again just use the C fopen command to open a file and now the fputs to put the string. Kotlin makes this even more convenient for interacting with the C library. There was no need to make a byte array native buffer we can just pass a Kotlin string directly to it and voila it works. At first I thought this would fail on larger array buffers. In testing I was able to allocate and write String arrays up to 275 MB on a system with 2 GB of physical memory and up over 1 GB on a system with 8GB of memory. That is plenty big enough for most uses. For people with larger blocks of data they’d probably end up breaking that down into individual lines anyway. A comparable method for writing an array of strings to a file is here:

fun writeAllLines(filePath:String, lines:List<String>, lineEnding:String="\n") {
    val file = fopen(filePath, WRITE_MODE) ?: 
        throw IllegalArgumentException("Cannot open output file $filePath")
    try {
        memScoped {
            lines.forEach {
                if(fputs(it + lineEnding, file) == EOF) {
                    throw Error("File write error")
            }
        }
    } finally {
        fclose(file)
    }
}

So with these three functions you have all you need to read and write text files easily in a Kotlin Native program. I’ll be going over an example of doing this with CSV and JSON data in another post to show how seamlessly the native code operates with regular Kotlin code.



Picture of Me (Hank)

Categories

Updates (125)
Journal (118)
Software Engineering (111)
Daily Updates (84)
Commentary (67)
Methodology (58)

Archive

2020
2019
2018
2017
2016
2015
2014
2013