GPandas - Acclimating Python Data Scientists to Golang Ecosystem

Disclaimer - I am a python developer myself and this is a friendly fire article. Please don’t read it if you don’t want to better yourself or get offended easily.

Let’s admit it. Majority of data scientists have jumped onto the python bandwagon and are too afraid to face their fears of the unknown. Everyone likes to say they are “language agnostic” without understanding the true meaning of what it means.

A programming language is like a natural, human language in that it favors certain metaphors, images and ways of thinking.

• Seymour Papert

Being language agnostic doesn’t mean we pick up a random language to code everytime we solve a problem, rather it means the programmer selects the best tool available to them at that given moment that fits the use case perfectly. Code agnosticism is more about being good at one language and comfortable enough to convert the business logic to other language.

What language agnosticism is -

1. Being good at one language and able to deduce language patterns in other languages.
2. Able to convert feature from one language to other (duh)
3. Be able to survive error code on error code when transpiling into a language you are not familiar with.

What language agnosticism is not -

1. Using LLM or other tools to transpile code.
2. Thinking you are good at all programming languages… Understanding the niche features of a language takes getting your hands dirty.
3. Use your favorite language for everything (JS/Python for everything)

Why language agnosticism all of a sudden?

The topic of language choice came up unexpectedly during a data science team standup. Our MD asked what programming languages we were using for production code. Being right there, I answered first, listing my usual suspects: “Python and Golang.” (Mentioning my C and Rust experience seemed likely to unnecessarily complicate the conversation about our core stack.)

I felt pretty good about covering the bases, but the MD’s response took me by surprise. He stated, “As a developer, you should be language agnostic; you shouldn’t tell which language you code in.” Now, while I appreciate the spirit of choosing the right tool for the job (which is how I interpret true language agnosticism, as I defined earlier), this particular phrasing seemed… interesting. It felt like it slightly missed the mark on the practicalities of specializing and communicating your expertise.

Before I could process it further, the rest of the team quickly sensed the desired answer and chimed in with enthusiastic agreement about being “language agnostic.” Let’s just say, based on our day-to-day work, most of us knew this translated roughly to “Proficient in Python, can probably google some HTML if needed.” It was one of those moments that highlighted the gap between management ideals and developer reality, and made me realize the shortcomings of python and how my team maneuvers around it rather than switching to another language to solve the problem in hand.

The Managerial Side of Things

Following that standup, a conversation with my manager brought a critical point into focus: I was effectively the sole Golang practitioner within the team, possibly the company. From a team lead or management perspective, this presents a classic “bus factor” risk. Relying on a single individual for a specific technology stack creates a bottleneck and a potential point of failure if that person is unavailable. It’s a valid concern; unmanaged diversification in tooling can introduce significant maintenance overhead and operational risk.

This realization shifted my thinking. While I was keen to leverage Go’s advantages for certain problems, simply introducing it wasn’t sustainable without broader team adoption. The challenge became: how could we strategically introduce Golang where it offered clear benefits, without creating an isolated silo? How could we lower the barrier to entry for a team deeply invested in the Python ecosystem? This wasn’t just about using my preferred tools; it was about finding a pragmatic way to enhance our team’s capabilities for the long term.

This led directly to analyzing the current landscape: Why the overwhelming reliance on Python, despite its known limitations in certain areas? And conversely, what were the real barriers preventing the adoption of a potentially more performant tool like Golang for specific use cases?

Python: The Lingua Franca of Data Science

So why do data scientists use python exclusively. It’s not that they are scared of data types, even though it’s complicated. It’s not the developer’s fault when Python throws a dtype(‘O’) and lets the interpretation of that datatype up to the viewer like it’s a classic Nolan film ending. Also, Python developers appreciate strong typing – especially the strongly worded TypeError they get at runtime.”

It all boils down to a few things I will list below that I have personally experienced with writing python vs something beautiful like rust or golang.

1. Python is messy. Yes sir, dump all the business logic into a jupyter notebook and click run all cells like a lunatic. I remember once when one of the interns got surprised when they got to know that you can wrap execution logic into a main() function and use if __name__ == '__main__': to execute it. Forget saving the planet, the most urgent environmental crisis for developers is global namespace pollution. One more globally declared temp variable and the whole ecosystem might just undefined itself.

2. Python is interpreted. Yes I am aware of the fact that we can run compiled language using a debugger to simulate that effect, but with that logic I can remove two tyres from my car and call it a bike. Python allows developers to run code line by line out of the box, making it perfect for debugging and tinkering with your code without losing the session state. Try that in golang and you will be waiting for that network call to process forever each time you compile. There are amazing projects like yaegi  from traefik  but last time I checked, it has 140+ issues, mainly related to unexpected behavior with reflections and method references getting altered (uh oh).

3. Less to type - My personal reason. I don’t want to write public static void func(String[] args) or something similar every time I want to write a function to append something to a string or make a DB call. The shorthands in python are plentiful and pleasant to use, like list comprehension, dict comprehension, iterator support like for-in loop, lambda functions and the list goes on and on.

The computer understands the same if we type less or more. If we choose to make it simpler for the developer then it becomes harder for the compiler and wise versa. It is the curse of abstraction.

4. Support in Data Science / AI field is superior - If I face an error in an obscure pandas ripoff package (self burn), there’s a chance my question in stack overflow/ reddit / source repo issues will ever see the light of the day. Instead, if I face an error in pandas, I will get vilified by Stack Overflow community for my “repeated” or google-able question. But there is a good chance I will receive support from the community even for the most obscure of errors, though I lose my aura in the process. I am not including the number of packages already built on python like scikit-learn, tensorflow and pytorch as python is an older language and golang would eventually catch up with enough developer interest and support.

The GIL-ty Conscience of Data Scientists

So if python is all flowery, where is the stench coming from? It’s coming from the python developer sweats who are trying to prove that they can write a function in 5 seconds and then it takes 10 minutes of waiting for it to do a database join. But realistically, here are the pitfalls of Python every python developer faces.

1. The GIL Curse - The GIL is essentially a mutex ensuring that only one thread executes Python bytecode at any given time within a single process, even on multi-core processors. While it simplifies memory management in CPython (the standard Python implementation), it effectively neuters true CPU-bound parallelism using standard Python threads. For data science, this can be a significant bottleneck. Many data processing tasks, complex simulations, or custom algorithm implementations are CPU-intensive. While core libraries like NumPy, Pandas, and Scikit-learn often cleverly sidestep the GIL by dropping down to C/C++/Fortran extensions for the heavy lifting, any significant amount of pure Python code trying to leverage multi-threading for speed gains on multiple cores will hit this wall. You might spawn multiple threads, but the GIL ensures only one is really working on Python code at a time. This forces developers into using the multiprocessing module (which has its own overheads and complexities) or relying entirely on optimized libraries, limiting the performance potential for custom, parallel Python logic compared to languages like Go or Rust designed for true concurrency from the ground up. The GIL can now be turned off as an experimental feature in Python 3.13, but it can lead to undefined behavior in legacy code, making it a design choice of future python developers.

2. Loosely Typed Language - Python being loosely typed is a double edged sword. Making it strongly typed requires extra effort from the developer.

assert isinstance(<variable>, <type>), f"type error: expected type for <variable> is 
<type>, <variable> is currently a {type(<variable>)}"

This is our go to type checker in python, but adding it after every important checkpoint in the code could be verbose or confusing to beginner python programmers. Expecting a data type and receiving something else can be detrimental in python, where multiple functions associated to different datatypes/ modules have similar names. This would lead to unexpected behavior at runtime down the road when something triggers the dormant bug.

3. Performance Issue: As seen in the graph below, CPython always comes last in raw performance. PyPy usually does it better, which makes use of a JIT compiler to achieve the performance boost. Go and Rust are always in the top 10 in performance metrics. It makes sense to use python in scripts, prototyping and non time critical things like backups and cron jobs. Definitely not for time sensitive data pipelines, high frequency/throughput data and type safety.

With all these pitfalls, it definitely makes sense to switch to golang or a compiled language for performance optimization. You can’t be using python or javascript and go around optimizing for a few milliseconds here and there. According to me, every python developer should know a compiled language along with python to create modules and also as a safety net in this uncertain market.

Golang: The Performance Fix Data Science Didn’t Know It Needed?

Alright, so Python’s got its issues, especially when your code takes longer to run than it took to write. If the GIL is giving you night sweats and dynamic typing feels like playing Russian roulette with your data pipelines, maybe it’s time to look at what Golang brings to the table. It’s not just about appeasing language-agnostic managers; there are actual, tangible reasons why Go might be the breath of fresh (and fast) air your data science workflow needs.

1. Speed That Doesn’t Require Selling Your Soul (or waiting forever): Remember that performance graph? Yeah, Python wasn’t exactly winning medals. Golang, being compiled, doesn’t mess around interpreting code line-by-line while you sip coffee. It builds a lean, mean, executable machine before you run it. For those hefty data transformations, complex simulations, or API services serving models that need to respond this century, Go’s raw speed can turn those 10-minute database joins into something far less painful. No more blaming the hardware when your Python script is chugging along like it’s running on a potato.

2. Concurrency Without the Headache: Forget the GIL’s one-thread-at-a-time nonsense. Golang was built for doing many things at once using goroutines – think lightweight threads that don’t cost an arm and a leg in overhead. Need to process multiple data streams, run parallel computations, or handle thousands of concurrent requests to your model API? Go handles it gracefully without forcing you into the clunky multiprocessing workarounds Python often demands. It actually uses those fancy multi-core processors you paid for.

3. Type Safety That Catches You Before You Fall: Tired of those lovely runtime TypeError surprises that pop up hours into a job, just because a function returned None instead of a float? Golang’s static typing forces you to be explicit upfront. The compiler acts like that annoyingly meticulous colleague who catches your mistakes before they cause a production fire. It means less time debugging weird type mismatches and more confidence that your data is actually the type you think it is. Yes, it means typing a bit more than Python’s free-for-all, but it might just save your sanity (and your weekend).

4. Surprisingly Gentle Learning Curve (for a compiled language): Okay, “compiled language” might trigger PTSD flashbacks to C++ templates or Rust’s borrow checker. But Go is different. It prides itself on simplicity and readability. The syntax is clean, there are fewer keywords, and it avoids much of the complex object-oriented hierarchy you find elsewhere. While it’s not quite as forgiving as Python, many Pythonistas find the transition smoother than expected. Think of it as Python’s slightly stricter, much faster cousin – you’ll recognize some family resemblance.

5. Deployment That Isn’t a Science Project: Ever tried deploying a Python app and ended up in dependency hell, wrestling with virtual environments and conflicting package versions? Go typically compiles everything down into a single static binary. Plop that file onto a server, and it just runs. No need to install Python, pip, or a specific set of libraries. For containerizing models or building data services, this simplicity is a massive win.

So, Golang clearly addresses many of Python’s performance and reliability pain points, offering a path to graduate from “it works on my machine” to “it works, fast, everywhere.” But asking a data science team steeped in Python to abandon the familiar comforts, especially the ubiquitous Pandas library, is a tough sell. How could we harness Go’s power for performance-critical tasks without completely losing that Pythonic data manipulation feel they rely on?

Inspiration for GPandas

GPandas is a high-performance data manipulation and analysis library written in Go, drawing inspiration from Python’s popular pandas library. It provides efficient and easy-to-use data structures, primarily the DataFrame, to handle structured data in Go applications.

This is the formal definition in the README.md of the project. I was aiming to bring all the goodness and interoperability of pandas to golang. What I realized when talking to my fellow python coders was the sheer dependence on abstraction to get their work done. Database operations - go with ORMs as validating SQL typing can get too tedious (it is not).

Abstraction is a necessary evil as it reduces the complexity exposed to the developer but in return it takes away performance and control. Abstraction of types was done using empty interfaces (ie any/comparable type), making the code look eerily similar to its pandas equivalent. The function signature was also kept similar to provide a sense of familiarity to developers migrating from python to golang.

For example

Pandas (Python)

import pandas as pd
import time
 
def main():
    start = time.perf_counter()
    _ = pd.read_csv('./customers-2000000.csv')
    end = time.perf_counter()
    print(end-start)
 
if __name__ == '__main__':
    main()

GPandas (Golang)

package main
 
import (
	"fmt"
	"gpandas"
	"time"
)
 
func readcsv() {
	start := time.Now()
	gp := gpandas.GoPandas{}
	_, err := gp.Read_csv("./customers-2000000.csv")
	if err != nil {
		fmt.Printf("Error reading CSV: %v\n", err)
		return
	}
	elapsed := time.Since(start)
	fmt.Printf("%f\n", elapsed.Seconds())
}

By understanding a few language patterns in Golang like error propagation and := operator for automatic type inference, one would realize the similarity in the abstraction present in both codes essentially enables the user to switch between the two languages.

General Features of GPandas

I won’t drag on for long as this article has become longer than anticipated. The core idea behind GPandas was to provide a familiar yet performant data manipulation experience within the Go ecosystem. Think Pandas, but leveraging Go’s strengths. Here’s a quick rundown of what it offers (or aims to offer):

• The DataFrame, Reimagined in Go: At its heart, GPandas provides the DataFrame – that familiar two-dimensional, labeled data structure with columns of potentially different types. It’s designed for efficiency, leveraging Go’s static typing and memory layout.

• Getting Data In: Just like its Python inspiration, GPandas supports reading data from common formats. Currently, you can load data directly from CSV and JSON files into DataFrames, making it easy to integrate with existing data sources.

• Slicing, Dicing, and Filtering: Need to select specific rows or columns? Filter data based on conditions? Sort your results? GPandas provides methods for these fundamental data selection and manipulation tasks, mirroring many common Pandas operations.

• Grouping and Aggregating: The powerful GroupBy functionality, essential for split-apply-combine workflows in data analysis, is part of the package. You can group data based on column values and perform aggregate calculations.

• Joining Forces (Merging Data): Combining data from different sources is crucial. GPandas includes capabilities for merging and joining DataFrames based on common columns or indices.

• Handling the Gaps: Real-world data is messy. The library includes tools for detecting and handling missing data (NaN values), allowing you to fill them using various strategies.

• Performance Baked In: By building on Go, GPandas inherently benefits from compiled code speed and Go’s efficient concurrency model, aiming to outperform pure Python equivalents for many tasks, especially CPU-bound ones.

GPandas performed better than Python in all the initial benchmark tests I performed, such as IO operations (20%-35% improvement) and aggregation (15% improvement) for a csv file with 2 million rows.

Essentially, GPandas tries to lower the barrier for Python/Pandas users moving to Go for data tasks by providing a recognizable API while delivering the performance and reliability benefits of the Go language.

Future Direction of the GPandas

So, what’s the grand plan for GPandas? Well, it kicked off as just a personal project, something to scratch an itch and maybe help my team stop sweating over the GIL. Surprisingly, it pinged on the radar of the Golang subreddit – a corner of the internet I didn’t expect much from for this kind of thing. Suddenly, suggestions and feature requests started rolling in, actual useful ideas from fellow Go developers. I even managed to rope in a few individual contributors, though whether we’re building the future of Go data science or just a more complex way to read CSVs, only time will tell.

Let’s be honest, the real boss fight is adoption and fitting into the wider ecosystem. Python’s ecosystem is a juggernaut; Pandas plugs into scikit-learn, matplotlib, and countless other tools almost effortlessly. GPandas has to carve out its niche in Go’s much younger data science landscape, figuring out how (or if) it can achieve that kind of interoperability. That’s a monumental challenge, especially when competing with the sheer inertia of Python.

But hey, regardless of whether GPandas takes over the world or quietly gathers dust on GitHub, the whole process has been invaluable. Seriously, digging deep into Pandas internals (figuring out which knob opens which door, as the saying goes), battling Go’s type system to make DataFrames behave, and navigating the sometimes chaotic, sometimes brilliant world of open source collaboration – that’s the kind of learning you can’t get from just reading documentation. It’s been a ride, and the skills picked up along the way are worth it, no matter the project’s final destination.