In this blog post we will dive down in to the difference between push based vs pull based query engines. As simple as is sounds push based is based upon that data is pushed from the sink through the different operators, this is used by snowflake and argued to be superior for OLAP which we will dive deeper into. Pull based have been around for a longer time and is based upon that data is pulled from the sink up through the operators, this is also known as theVolcano Iterator Model…

This blog post will try to dive down in to apache arrow and specifically the Go memory allocation for Apache arrow. Apache arrow state that they allow for the following types of memory allocations: Go default allocations(standard go GC collected memory) CGo allocator(memory allocated through CG0) Checked Memory Allocator Will deep dive in to these once in a follow up blog post. Today the goal is to extend with a new memory allocator, mostly becuase I read up on go memory arena which where introduced in to 1.…

Applying for jobs are fun, but it is not fun to write cover letters at least not compare to hacking on some new open source tool or testing out something new. Therefore the goal is to reduce the time to apply for a job thus the goal today is to generate cover letters. For this we will need the following: CV, I will copy paste mine from linkedin(minimal effort and I spend way to much time on linkedin so it is up to date) Job add(will take one from linkedin that sounds interesting) Get the CV.…

This blog post is a collection of awesome go resource and will be continuously update. The goal is that each resource should be describe shortly as well. Blogs: Generics can make your Go code slower Effective go Go memory model inline Videos: Obscure Go Optimisations A Guide to the Go Garbage Collector …

In this blog post, we will discuss how to store Kubeflow Pipeline templates in GCP Artifact Registry, enabling reusability and version control for your pipelines. Using Artifact Registry over Cloud Storage simplifies version control and allows for easier collaboration between single or multiple users. The Kubeflow Pipelines SDK registry client is a new client interface that you can use with a compatible registry server (ensure you are using the correct KFP version), such as Artifact Registry, for version control of your Kubeflow Pipelines (KFP) templates.…

This blog post will go over how to build and run your very first kubeflow pipeline (kfp). In short, Kubeflow Pipelines is a platform for building and deploying portable, scalable machine learning (ML) workflows based on Docker containers. There are a lot of possibilities to run the pipelines, but in this series, we will use gcp vertex pipelines. Vertex will be the runner, but the pipelines will follow the kubeflow conventions meaning you can run them on whatever platform at hand or host kubeflow on your own Kubernetes cluster.…

This blog post provides a step-by-step guide for setting up your VS Code dev container to work with Google Cloud Platform (GCP) services and APIs by configuring default GCP credentials. By authenticating your application with your GCP credentials, you can access the necessary resources without requiring additional authentication steps, saving time and streamlining your development workflow. Configuring Default GCP Credentials To use default credentials with GCP, you can follow the steps below:…

VS docker Dev Container Are you tired of dealing with messy, inconsistent development environments that slow down your workflow? Look no further than VS Code dev containers, a powerful feature in Visual Studio Code that can streamline your development process and improve consistency across different environments. In this post, we’ll dive into what dev containers are and how you can use them to supercharge your development workflow. We’ll also walk through how to set up a dev container for a Python environment and you will get an understsnding how easy it is to get started.…

This blog post will dive into building postgres extensions using pgx in rust. In order to do something existing we will ride the hype curve and integrate ChatGPT3 into postgres. TLDR: repo Setup First step is to set up pgx: $ cargo install --locked cargo-pgx $ cargo pgx init We will then create a new create using: $ cargo pgx new my_extension $ cd my_extension Which should give you something like:…

As part of the go 1.20 release memory areas where introduced to the standard lib but not mentioned in the release notes but is still being discussed as of 2023-03-02 here. Memory arenas allow users to allocate memory and are described by the docs as: The arena package provides the ability to allocate memory for a collection of Go values and free that space manually all at once, safely. The purpose of this functionality is to improve efficiency: manually freeing memory before a garbage collection delays that cycle.…