While migrating data for ipythonblocks.org I ran into an issue that stumped me for about a day: I could read data from my database after the migration, but not insert anything. When I tried, Postgres complained that there I was trying to insert a duplicate primary key. The twist, though, was that I wasn’t specifying a primary key, I expected Postgres to automatically generate and insert a new, valid key. What I ultimately discovered was that because the data I had migrated already contained primary key values, the Postgres sequence created for the table primary key column had not been advanced to the point where it would generate new key values that weren’t already in the table. (A sequence is a stateful number generator kind of like Python’s range.) I had to manually set the sequence state using Postgres’ sequence manipulation functions. This post contains a demonstration of this problem and the solution. SQL was run against Postgres 9.6.3 using pgcli. Continue reading “Setting the State of a Postgres Sequence”
Month: July 2017
ipythonblocks.org Move: Part 4 – Application Updates
This is Part 4 in a series of blog posts describing my move of ipythonblogs.org from Rackspace to Heroku. In this post I’ll describe the updates I’ve made to the application layer of ipythonblocks.org. Other posts are:
- Part 1: Introduction and Architecture
- Part 2: Data Migration
- Part 3: Database Interface Updates
- Part 4: Application Updates
The application logic is not really changed in this update, the bulk of changes are to support providing SQLAlchemy sessions to allow database access during requests. (See Part 3 for discussion of the database interface layer of ipythonblocks.org.)
Application Overview
ipythonblocks.org is powered by Tornado, which combines an application framework, web server, and asynchronous features. On Heroku the application is started with the command
python -m app.app --port=$PORT --db_url=$DATABASE_URL
$PORT and $DATABASE_URL are environment variables provided by Heroku to specify, respectively, which port to listen on and where to find the Postgres database attached to the app. These are parsed from the command line by Tornado’s options module and made available globally on the tornado.options.options variable. Continue reading “ipythonblocks.org Move: Part 4 – Application Updates”
ipythonblocks.org Move: Part 3 — Database Interface
This is Part 3 in a series of blog posts describing my move of ipythonblogs.org from Rackspace to Heroku. In this post I’ll describe the updates I’ve made to the database interface module of ipythonblocks.org. Other posts are:
- Part 1: Introduction and Architecture
- Part 2: Data Migration
- Part 3: Database Interface Updates
- Part 4: Application Updates
The big change to the database interface module was the switch from dataset to SQLAlchemy for database abstraction. This involves using the ORM models described in Part 2, removing the JSON de/serialization functions needed to use SQLite, removing use of memcached, and updating tests to use a Postgres database to match production. The full diff is here, but I’ll breakdown the important points below. Continue reading “ipythonblocks.org Move: Part 3 — Database Interface”
ipythonblocks.org Move: Part 2 – Data Migration
This is Part 2 in a series of blog posts describing my move of ipythonblogs.org from Rackspace to Heroku. In Part 1 of this series I described my motivation for the move and the broad changes I expect to make as part of the migration. In this post I’ll describe the grid data model and how I migrated the existing grid data from SQLite to Postgres. Other posts are:
- Part 1: Introduction and Architecture
- Part 2: Data Migration
- Part 3: Database Interface Updates
- Part 4: Application Updates
Continue reading “ipythonblocks.org Move: Part 2 – Data Migration”
ipythonblocks.org Move: Part 1
This is Part 1 in a series of blog posts describing my move of ipythonblogs.org from Rackspace to Heroku. In this first post I’ll describe the existing deployment and what I intend to change during the migration. Other posts are:
- Part 1: Introduction and Architecture
- Part 2: Data Migration
- Part 3: Database Interface
- Part 4: Application Updates
As a side project I maintain a Python library called ipythonblocks that displays colored grids in a Jupyter Notebook. (See also this intro blog post.) It can be useful for teaching or for a bit of fun art. I also maintain the website ipythonblocks.org that allows users to post their ipythonblocks grids on the internet to be shared. ipythonblocks.org has been hosted on Rackspace since I first launched it, but now I’m migrating the site to Heroku for easier maintenance and deployment. Continue reading “ipythonblocks.org Move: Part 1”
Pen and Pants 