Notes

Atom Feed

Upgrading MariaDB Database Versions

June 2, 2024

This script updates the database version for MariaDB docker containers. Given the container name, it will stop the container and launch a new MariaDB container with the updated version of MariaDB but with the original data and network.

Notes:

  • The script requires the name of the docker container as an argument
  • The new docker container has some typical recommended configurations set like --cap-add=sys_nice to optimize cpu scheduling.
  • The script will prompt for the root password in order to execute upgrade scripts
#!/bin/bash

set -exuo pipefail
IFS=$'\n\t'

BASEDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
cd "$BASEDIR/.." || exit 1

NAME="$1"
VERSION="11.4.2"
VOLUME="$(docker container inspect "$NAME" | jq -r .[0].Mounts[0].Name)"
NETWORK="$(docker container inspect "$NAME" | jq -r '.[0].NetworkSettings.Networks | keys[0]')"

echo "$VOLUME"

docker pull "mariadb:$VERSION"
docker stop "$NAME" || true
docker rm "$NAME" || true
docker run \
    --detach \
    --restart=always \
    --network="$NETWORK" \
    --name="$NAME" \
    --volume "$VOLUME:/var/lib/mysql" \
    --cap-add=sys_nice \
    "mariadb:$VERSION" \
    --character-set-server=utf8mb4 \
    --collation-server=utf8mb4_unicode_ci \
    --binlog_expire_logs_seconds=3600
docker exec -it "$NAME" mariadb-upgrade -u root -p --force

Permalink

Concurrent Python Example

January 1, 2024

Reference example of using concurrent.futures in Python >3.2. Standard parallel programming warnings still apply (race conditions, sharing state, inter-process communication):

import concurrent.futures
import time


def calculate(limit: int) -> int:
    """ An example function that takes a long time to run """
    x = 1
    for i in range(1, limit):
        x = (x + i) % 10
    return x


def iterate(data: list[int]) -> list[int]:
    """ A base case without concurrency """
    return [calculate(i) for i in data]


def concurrent_submit(data: list[int]) -> list[int]:
    """ Concurrency using executor.submit """
    result: list[int] = []
    with concurrent.futures.ProcessPoolExecutor() as executor:
        futures: list[concurrent.futures.Future[int]] = []
        for i in data:
            future = executor.submit(calculate, i)
            futures.append(future)
        for future in futures:
            result.append(future.result())
    return result


def concurrent_map(data: list[int]) -> list[int]:
    """ Concurrency using executor.map """
    with concurrent.futures.ProcessPoolExecutor() as executor:
        data = list(executor.map(calculate, data))
    return data


def main() -> None:
    """ Execute each of the implementations and time it """
    data = [10**7 + x for x in range(20)]
    print(data)

    """
    # Non-parallel implementation
    start = time.time()
    output = iterate(data)
    duration = time.time() - start
    print('iterate: %f' % duration)
    print(output)
    """

    # Parallel with executor.submit()
    start = time.time()
    output = concurrent_submit(data)
    duration = time.time() - start
    print('concurrent_submit: %f' % duration)
    print(output)

    # Parallel with executor.map()
    start = time.time()
    output = concurrent_map(data)
    duration = time.time() - start
    print('concurrent_map: %f' % duration)
    print(output)


if __name__ == '__main__':
    main()

Permalink

Updating UUIDField on MariaDB to Django 5

December 27, 2023

Updating from Django 4 to Django 5 comes with an incompatible change to how Django UUIDFields are stored in MariaDB databases - in Django 4, Django would store UUIDFields as char(32) types but in Django 5, Django would store UUIDFields as uuid types. However, the Django 5 upgrade notes are incomplete. Following the directions on a nontrivial Django project, it’s still likely that you’ll get database errors like django.db.utils.OperationalError: (4078, "Cannot cast 'int' as 'uuid' in assignment of test_database.table_column.id") and errors from fitting 36-character UUID strings into 32-character database fields.

In order to fix this problem, I found that it’s better to convert char(32) MariaDB fields into uuid fields first before migrating to Django 5. That way, there are fewer changes during the update. In order to do so,

  1. While still on Django 4, replace django.models.UUIDField with a uuid field using a uuid database type with:

    # app/models.py
    
    from django.db import models
    
    
    class RealUUIDField(models.UUIDField):
        def db_type(self, connection):
            return "uuid"
    
    
    class Model(models.Model):
        id = RealUUIDField(primary_key=True, default=uuid.uuid4, editable=False)
        # Formerly:
        # id = models.UUIDField(primary_key=True, default=uuid.uuid4, editable=False)
    
  2. Explicitly declare older migrations on the uuid field as a char(36) field. Note that this needs to be char(36) because Django will attempt to store UUIDs with dashes ('06cb5e67-467f-4675-91f3-ca466bcee805' instead of '06cb5e67467f467591f3ca466bcee805'). In migration files, replace models.UUIDField with:

    # app/migrations/0001_migration.py
    class Char36UUIDField(models.UUIDField):
        def db_type(self, connection):
            return "char(36)"
    
    class Migration(migrations.Migration):
        operations = [
            migrations.CreateModel(
                name='Model',
                fields=[
                    ('id', Char36UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
                    # Formerly:
                    # ('id', Char36UUIDField(default=uuid.uuid4, editable=False, primary_key=True, serialize=False)),
                ],
            ),
        ]
    
  3. Run ./manage.py makemigrations to generate a migration that will convert the database column from a char(32) to a uuid type.

  4. Install the django==5.0 pip package. Fix any other incompatibilities.

  5. Replace RealUUIDField back to models.UUIDField.

Permalink

Replacing Setup.py

December 7, 2023

Setup.py is more or less deprecated and should be replaced by alternatives like pyproject.toml or poetry. This requires installing a few dependencies:

pip install build twine
setup.py command alternative command
setup.py sdist python -m build
setup.py bdist_wheel python -m build
setup.py test python -m unittest
setup.py install pip install .
setup.py develop pip install -e .
setup.py upload twine upload
setup.py check twine check

Permalink

Fixing Mariadb --Column-Statistics Errors

June 5, 2023

Recent versions of mysqldump include a config to include a COLUMN_STATISTICS table. While StackExchange answers recommand passing a --column-statistics=0 flag to mysqldump, this isn’t always possible, e.g. when some code like Django is running mysqldump for you.

Instead of patching your mysqldump calls, if you recently upgraded from using mysql to mariadb, you should check if you’re still using the Oracle mysql version of mysqldump. Switching to the mariadb version of mysqldump may solve your problems. On ubuntu/debian, you can do so with

apt update
apt remove mysql-client
apt install mariadb-client

Permalink

Geographic Geometry Simplification

February 2, 2023

Linters

January 2, 2023

Installing Mysqlclient in Python Slim Docker Image

December 29, 2022

Processor Trends

December 19, 2022

Resizing a Ubuntu Disk in a UTM VM

October 19, 2022

Bash File Test Operators

October 5, 2022

Python Generic Type Annotations

May 28, 2022

Mac Menubar Applications

February 17, 2022

Logodust

February 13, 2022

Python Releases

January 8, 2022

Debian Releases

January 7, 2022

Ubuntu Releases and Support Periods

December 18, 2021

Monitoring System CLIs (Top for X)

December 18, 2021

Fixing "EFI stub: Exiting boot services and installing virtual address map..."

December 11, 2021

ARM Support

November 29, 2021

Map Caps Lock to Escape for Vim

November 24, 2021

Download and Convert Youtube Playlists to MP3 Files

July 15, 2021

Nobody Ever Got Fired for Copying FAANG

June 27, 2021

Removing Token Authentication From Jupyter/iPython Notebooks

May 31, 2021

Debian and Ubuntu Releases

February 13, 2021

Setting Up FastAI Fastbook on a Fresh Ubuntu Instance

January 31, 2021

Tip for Developer Tools Startups

January 30, 2021

A Better Go Defer

October 20, 2020

Covid-19 Economy Predictions

October 13, 2020

Basic Docker Monitoring

July 4, 2020

Switching From Go Dep to Go Mod

May 30, 2020

Upgrading LibMySQLClient in Python MySQLDB/MySQLClient

May 25, 2020

Developing Django in Production

May 15, 2020

Quote

March 5, 2020

Sendmail Wrapper for Mailgun

March 1, 2020

Python Release Support Timeline

December 26, 2019

Use the Default Flake8 Ignores

December 14, 2019

Making Pip Require a Virtualenv

December 5, 2019

Engineering Toolbox

November 30, 2019

Node Timezones

November 1, 2019

Sampling Samples

August 21, 2019

Rotating a NxN Matrix in One Line of Python

July 27, 2019

iTerm2 Search History

July 19, 2019

Nginx Auth With IP Whitelists

June 29, 2019

Bash Strict Mode

May 11, 2019

Optimizing Asus Routers for Serving Websites With Cloudflare

May 5, 2019

Browserify, Mochify, Nyc, Envify, and Dotenv

April 1, 2019

Scraping Images From Tumblr

February 24, 2019

There Are Too Many NPM Packages

February 10, 2019

Programmers Writing Legal Documents

January 31, 2019

Solidity Review

November 17, 2018

Likwid

November 9, 2018

My First Server's IP

November 9, 2018

Installing Netdata

September 23, 2018

Interrobang Versus Shebang

July 10, 2018

Bad Interview Questions

July 8, 2018

Showing Users in Different Databases

July 7, 2018

Some MIT (Undergraduate) Admissions Interview Advice

July 4, 2018

Optimize the Develop-Test-Debug Cycle

April 22, 2018

Example of Python Subprocess

March 23, 2018

Spotted in Taiwan

January 20, 2018

Fixing "Fatal Error: Python.h: No Such File or Directory"

December 16, 2017

Cassandra Primary Keys

December 11, 2017

MyPy Review

November 2, 2017

Griping About Time Zones

October 26, 2017

Bundling Python Packages With PyInstaller and Requests

September 23, 2017

Go Receiver Pointers vs. Values

September 4, 2017

Fixing statsonice.com Latency

September 1, 2017

Showing Schemas in Different Databases

August 26, 2017

Straight Lines

June 2, 2017

Emerson on Intellect

May 29, 2017

Core Metric for Developer Productivity

May 21, 2017

How to Capture a Camera Image With Python

May 7, 2017

Python Has a Ridiculous Number of Inotify Implementations

May 2, 2017

Projects: Gentle-Alerts

April 27, 2017

Creating a New PyPI Release

April 24, 2017

Eva Air USB Ports

April 24, 2017

Projects: Git-Browse

March 18, 2017

Cassandra Compaction Strategies

March 5, 2017

Code Is Like Tissue Paper

January 25, 2017

Seen in a Bathroom Stall at MIT

January 24, 2017

Underused Python Package: Webbrowser

January 21, 2017

Pax ?

January 5, 2017

Golang Review

January 2, 2017

Wadler's Law

December 15, 2016

Tunnel V2

December 8, 2016

MultiPens

December 5, 2016

SSH Tunnel

September 18, 2016

That Time I Was a Whitehat Hacker

September 18, 2016

Comparison of Country and Company GDPs

September 8, 2016

Sketching Science

September 8, 2016

Tech Hiring Misperceptions at Different Companies

July 22, 2016

Calculating Rails Database Connections

June 26, 2016

DevOps Reactions

June 12, 2016

Tuning Postgres

June 9, 2016

Fibonaccoli

June 4, 2016