Siva Mahadevan

I/O Idioms in the Iversonian Array Languages

2024-06-04

I/O Idioms in the Iversonian Array Languages | Siva's Blog

I/O Idioms in the Iversonian Array Languages

Around three years ago, I came across a post on Hacker News announcing the first episode of The Array Cast, a podcast about the array programming languages. This post caught my eye quite quickly because I had never heard of the array programming paradigm before and wanted to learn more about it. I was working for NVIDIA at the time, so I was (and still am) enthralled by the performance gains of restructuring code to take advantage of large-scale parallelism. Upon first glance, I understood it as a way to express homogeneous computations over a collection of data without explicitly writing housekeeping code to loop over the collection. It registered in my mind as a SIMD-first (Single Instruction, Multiple Data) paradigm.

Hacker News: The Array Cast – A podcast about the array programming languages

Life got in the way of spending quality time doing a proper deep-dive into the paradigm. I decided to first learn the J language after coming across a nice book called “J for C programmers”, which was definitely applicable to me. I took advantage of a long flight to read through a good chunk of the book, stopping after the “Input and Output” section. It was a good experience for me because the ASCII symbols were easy to type in a normal text editor and the wiki was very detailed (even if it was quite dense) when I was in need of help. The concepts of abstracting out the loops and carefully constructing customized “views” over raw multidimensional data became very attractive.

J for C Programmers

At this point, I started to follow the ArrayCast and listen to every episode as they came out. I really love it! The technical discussion interleaved with humour and stories from all the accomplished array programmers around the world is a great way to learn more about this powerful paradigm. As of now, I'm really interested in BQN and will be spending some free time reading through all the wonderful documentation.

The BQN programming language

A Mental Barrier

To someone who has been programming for 15 years now, learning the syntax and understanding the parsing/evaluation semantics of an array language is not too difficult. Discovering and internalizing the idioms that experienced programmers use to perform real-world tasks is the hard part. For example, as a POSIX shell enthusiast, I might be able to easily conjure certain blocks of code for chaining together a large pipeline of bite-sized string processing utilities to parse some arbitrary text data. Others might find it to be a daunting task to scour the POSIX specifications to find exactly the flags they are looking for to manipulate the data in a certain way. I currently feel like this with the array programming languages.

What I have noticed is that although the currently available documentation and pedagogical material is really high quality and quite captivating, it misses a lot of the idioms that newcomers might need to start writing programs with real-world data. I'm grateful for the existence of the “J for C programmers” book for this reason, since it does use a practical approach to appeal to procedural programmers like me who are used to consuming data in a certain C-oriented way. But what I'm really looking for (and this may be a bit lazy of me, but this is an unfiltered thought of mine) experienced array programmers to teach the best practices of:

consuming messy real-world data in unstructured plain text, structured binary, and RFC-standardized data interchange formats
representing unnecessarily sparse “structs” from other programming languages as dense arrays that are primed for easy manipulation using array language primitives
serializing array data into RFC-standardized data interchange formats suitable for consumption by external applications

This is my question to the panel of The Array Cast. As I am currently learning BQN, my question will be more tailored towards it, but I love the paradigm as a whole and all languages within it.

How do the Iversonian array languages like to interchange data with the outside world?

Conor's YouTube videos and podcast appearances in CodeRecursive and ADSP are a great source of content for algorithmic and combinatorial idioms in BQN using practical problem-solving techniques. These are great for solving isolated tasks such as the ones found in LeetCode, Project Euler, Advent of Code, and similar coding challenges.

Conor’s Youtube Channel

Marshall and BQN contributors’ documentation on the BQN website is extensive, articulate, and quite captivating with its high quality content and comedic references. The BQN source code is also full of incredibly high content pedagogical material on the topic of data interchange, such as the following:

A Markdown parser used to generate the beautiful website

A multithreaded Make-like build tool used to build CBQN

The BQN-libs project that provides various utilities for data interchange

Going through the last of the above points primitive-by-primitive should really help a newcomer to learn some idioms, and I have put this task on my TODO list. However, this prompts the question in the heading of this section. Hypothetically, in an ideal world where array programming languages are the “lingua franca” of computer programming, what on-disk and in-memory data interchange formats and data-organizational methods are IDEAL for array language consumption? I realize that this is a context-dependent and industry/domain-dependent question, but I observe a few patterns in the industry right now:

Flexible, but verbose formats like JSON are common for communicating data between web services due to first class citizenship in JavaScript and other web-oriented languages
Simple textual tabular data formats like {C,T}SV are common when rapidly iterating on data analysis in scientific and numerical computation
Protobuf and similar dense serialization formats are common for communicating between heterogeneous submodules within monolith projects
Binary formats are common in multimedia since the compression of data and ease of consumption by hardware implementations is the priority

These are all examples of widely used formats that are ripe for consumption by the Iversonian array languages. But I don't know where to start. I would absolutely love to listen to a discussion on this topic and learn what the best practices are in the eyes of all the panel members of The Array Cast. Thanks for the incredible discussions so far! I can't wait to listen to the upcoming episodes.

I’m Impressed By Poudriere

2024-02-09

I’m Impressed By Poudriere | Siva's Blog

I’m Impressed By Poudriere

When I add new software to my digital workflow, the first thing I do is learn how to customize it. This usually means scouring the internet for blog posts written by people who have successfully configured this software in a non-trivial way and implemented it into their long-running systems with stable uptimes. Recently, I came across a blog post by “vermaden” detailing a FreeBSD concept called “PkgBase”.

Personal FreeBSD PKGBASE Update Server

PkgBase on FreeBSD Wiki

PkgBase

To my understanding, “PkgBase” is a way to encapsulate FreeBSD’s base system (comprising both kernel and userland) into standard FreeBSD packages that can be installed normally using `pkg`. Normally, a system would consume a single `pkg` repository that contains packages from the Ports collection. With “PkgBase”, you can add a second `pkg` repository containing “PkgBase” packages. The entire system (base and ports) can be upgraded in a single shot using `pkg upgrade`.

I find this really elegant. A clear divide between FreeBSD’s official sources and external third-party sources is still maintained, but distributing and installing all of it can be done in a unified manner. It’s quite convenient for me to just run a few commands to upgrade my entire system.

My old way of doing things

So far, I’ve been taking a rolling-release approach to my FreeBSD build systems. This involves:

Building the base system from source on the `stable/$LATEST_MAJOR_VERSION` branch (stable/14 at the time of writing)
Installing the base system onto the same host using `make installkernel` and `make installworld`
Building ports from source using Poudriere and a custom ports list trimmed down to only the ports I use
Installing ports by pointing `pkg` to my local Poudriere repository and running `pkg install` or `pkg upgrade` accordingly

I’ll just quickly summarize the first two bullets here using a 10-step process paraphrased from the FreeBSD handbook:

# cd /usr/src                            (1)
# git checkout stable/14                 (2)
# git pull                               (3)

get up-to-date on the news by reading the UPDATING file

# make -j$(nproc) buildworld buildkernel (4)
# make installkernel                     (5)
# reboot                                 (6)
# etcupdate -p                           (7)
# make -C /usr/src installworld          (8)
# etcupdate -B                           (9)
# shutdown -r now                       (10)

How to build FreeBSD base from source

I’ve trimmed my configuration for each of these builds in this way:

A custom `src.conf` that contains a ton of `WITHOUT_xxx` options to remove stuff I don’t need
A custom `KERNCONF` that only keeps bare minimum support for my particular hardware
A custom ports list that only includes ports that I actually use
A native CPUTYPE for the target system
A custom `make.conf` that carefully sets global defaults for all port `OPTIONS` that are good for my needs

This results in a lean, mean, FreeBSD machine. I’m happy with the level of fat that this method burns off for my personal use.

The new way

However, the second step of “installing the base system” is annoying. It mostly works fine for installing a base system onto the same host, but what about managing tons of hosts with heterogeneous hardware? I’d ideally like to have a fleet of lean, mean, FreeBSD machines, with tidy custom configuration. This is when I noticed the bit near the bottom of vermaden’s blog post:

I did not (yet) [investigate] the Poudriere way of doing it – but [Mina Galić] was kind enough to share [their] HOWTO on the topic.

PkgBase Poudriere HOWTO

Perfect. After searching the internet more on this topic, I came across an upstream Poudriere discussion on the same topic.

How to use poudriere to (only??) build pkgbase?

The only thing I need to do is create a Poudriere jail with the `-B` option. This is so simple!

poudriere jail -c -j poudriere_stable14_upstream -b -m git+https -v stable/14 -B -K MYKERNCONF -z myconfig

Poudriere is incredible software

As mentioned, I’ve already been using Poudriere to create a personal Ports repository. What I learned from Mina’s post is that Poudriere can also be used to build a personal FreeBSD base repository.

This essentially means I can create my own carefully curated and trimmed personal FreeBSD distribution with just a single tool: Poudriere! This is incredible and I love this software. I’ll be writing more comprehensively in the future about how I use Poudriere to maintain my FreeBSD systems.

It seems that Poudriere has followed the UNIX philosophy perfectly. It really only does one thing, but it does it extremely well.