Setting the State of a Postgres Sequence

While migrating data for ipythonblocks.org I ran into an issue that stumped me for about a day: I could read data from my database after the migration, but not insert anything. When I tried, Postgres complained that there I was trying to insert a duplicate primary key. The twist, though, was that I wasn’t specifying a primary key, I expected Postgres to automatically generate and insert a new, valid key. What I ultimately discovered was that because the data I had migrated already contained primary key values, the Postgres sequence created for the table primary key column had not been advanced to the point where it would generate new key values that weren’t already in the table. (A sequence is a stateful number generator kind of like Python’s range.) I had to manually set the sequence state using Postgres’ sequence manipulation functions. This post contains a demonstration of this problem and the solution. SQL was run against Postgres 9.6.3 using pgcli. Continue reading “Setting the State of a Postgres Sequence”

Setting the State of a Postgres Sequence

ipythonblocks.org Move: Part 4 – Application Updates

This is Part 4 in a series of blog posts describing my move of ipythonblogs.org from Rackspace to Heroku. In this post I’ll describe the updates I’ve made to the application layer of ipythonblocks.org. Other posts are:

  • Part 1: Introduction and Architecture
  • Part 2: Data Migration
  • Part 3: Database Interface Updates
  • Part 4: Application Updates

The application logic is not really changed in this update, the bulk of changes are to support providing SQLAlchemy sessions to allow database access during requests. (See Part 3 for discussion of the database interface layer of ipythonblocks.org.)

Application Overview

ipythonblocks.org is powered by Tornado, which combines an application framework, web server, and asynchronous features. On Heroku the application is started with the command

python -m app.app --port=$PORT --db_url=$DATABASE_URL

$PORT and $DATABASE_URL are environment variables provided by Heroku to specify, respectively, which port to listen on and where to find the Postgres database attached to the app. These are parsed from the command line by Tornado’s options module and made available globally on the tornado.options.options variable. Continue reading “ipythonblocks.org Move: Part 4 – Application Updates”

ipythonblocks.org Move: Part 4 – Application Updates

ipythonblocks.org Move: Part 3 — Database Interface

This is Part 3 in a series of blog posts describing my move of ipythonblogs.org from Rackspace to Heroku. In this post I’ll describe the updates I’ve made to the database interface module of ipythonblocks.org. Other posts are:

  • Part 1: Introduction and Architecture
  • Part 2: Data Migration
  • Part 3: Database Interface Updates
  • Part 4: Application Updates

The big change to the database interface module was the switch from dataset to SQLAlchemy for database abstraction. This involves using the ORM models described in Part 2, removing the JSON de/serialization functions needed to use SQLite, removing use of memcached, and updating tests to use a Postgres database to match production. The full diff is here, but I’ll breakdown the important points below. Continue reading “ipythonblocks.org Move: Part 3 — Database Interface”

ipythonblocks.org Move: Part 3 — Database Interface

ipythonblocks.org Move: Part 2 – Data Migration

This is Part 2 in a series of blog posts describing my move of ipythonblogs.org from Rackspace to Heroku. In Part 1 of this series I described my motivation for the move and the broad changes I expect to make as part of the migration. In this post I’ll describe the grid data model and how I migrated the existing grid data from SQLite to Postgres. Other posts are:

  • Part 1: Introduction and Architecture
  • Part 2: Data Migration
  • Part 3: Database Interface Updates
  • Part 4: Application Updates

Continue reading “ipythonblocks.org Move: Part 2 – Data Migration”

ipythonblocks.org Move: Part 2 – Data Migration

ipythonblocks.org Move: Part 1

This is Part 1 in a series of blog posts describing my move of ipythonblogs.org from Rackspace to Heroku. In this first post I’ll describe the existing deployment and what I intend to change during the migration. Other posts are:

  • Part 1: Introduction and Architecture
  • Part 2: Data Migration
  • Part 3: Database Interface
  • Part 4: Application Updates

As a side project I maintain a Python library called ipythonblocks that displays colored grids in a Jupyter Notebook. (See also this intro blog post.) It can be useful for teaching or for a bit of fun art. I also maintain the website ipythonblocks.org that allows users to post their ipythonblocks grids on the internet to be shared. ipythonblocks.org has been hosted on Rackspace since I first launched it, but now I’m migrating the site to Heroku for easier maintenance and deployment. Continue reading “ipythonblocks.org Move: Part 1”

ipythonblocks.org Move: Part 1

Palettable 3.0 Released

I’m happy to announce the release of Palettable version 3.0. Palettable is a Python library that packages a variety of color palettes for use with matplotlib or really anywhere. Here’s the full diff since the last release.

This release includes a number of new palettes:

The new cmocean, matplotlib, and MyCarta palettes are created from data that contains 256 color points per palette. By default palettes are created with lengths 3-20 colors, but you can request longer ones via the get_map function. For example, to get the matplotlib Viridis palette with 200 color points:

In [2]: palettable.matplotlib.get_map('Viridis_200')

You can find Palettable on the web at:

P.S.: Here’s a little demo notebook.

Palettable 3.0 Released

Ally Work

As folks become more aware of the social justice challenges we face as a country and in our communities they often wonder how they can help. There’s not one really great answer because we all have different experience levels, means, axes of privilege, and available free time, but there are some common-sense “getting started” points. People with privilege who help social just causes are often called allies (but more on this word below), and this post is about how allies can learn how to help. A very short and incomplete TL;DR:

  • don’t center your own ego and feelings at the expense of others’
  • examine and change your own behavior so you don’t perpetuate bad habits
  • whole-heartedly listen to people with less privilege than you
  • educate yourself on the history and literature of the movements you are interested in
  • step up when you can

The following are several resources for people looking for specific things to think about and act on. This list is focused on diversity-in-tech, and especially gender diversity, so pointers to other resources would be great.

(A note about the word “ally”: there’s a real failing of language here because we use this noun to refer to privileged supporters of social justice causes, but the causes don’t need nouns, they need actions. You can’t do 10 supportive things and then have the label “ally” for life. It’s a process of doing what you can, when you can. This essay gives a great breakdown of the distinction: http://www.shakesville.com/2013/04/on-fixed-state-ally-model-vs-process.html.)

Note: This is based on a series of posts I’ve been putting together at work to educate coworkers on diversity and inclusion topics.

Ally Work