Setting the State of a Postgres Sequence

While migrating data for ipythonblocks.org I ran into an issue that stumped me for about a day: I could read data from my database after the migration, but not insert anything. When I tried, Postgres complained that there I was trying to insert a duplicate primary key. The twist, though, was that I wasn’t specifying a primary key, I expected Postgres to automatically generate and insert a new, valid key. What I ultimately discovered was that because the data I had migrated already contained primary key values, the Postgres sequence created for the table primary key column had not been advanced to the point where it would generate new key values that weren’t already in the table. (A sequence is a stateful number generator kind of like Python’s range.) I had to manually set the sequence state using Postgres’ sequence manipulation functions. This post contains a demonstration of this problem and the solution. SQL was run against Postgres 9.6.3 using pgcli. Continue reading “Setting the State of a Postgres Sequence”

Setting the State of a Postgres Sequence

ipythonblocks.org Move: Part 4 – Application Updates

This is Part 4 in a series of blog posts describing my move of ipythonblogs.org from Rackspace to Heroku. In this post I’ll describe the updates I’ve made to the application layer of ipythonblocks.org. Other posts are:

  • Part 1: Introduction and Architecture
  • Part 2: Data Migration
  • Part 3: Database Interface Updates
  • Part 4: Application Updates

The application logic is not really changed in this update, the bulk of changes are to support providing SQLAlchemy sessions to allow database access during requests. (See Part 3 for discussion of the database interface layer of ipythonblocks.org.)

Application Overview

ipythonblocks.org is powered by Tornado, which combines an application framework, web server, and asynchronous features. On Heroku the application is started with the command

python -m app.app --port=$PORT --db_url=$DATABASE_URL

$PORT and $DATABASE_URL are environment variables provided by Heroku to specify, respectively, which port to listen on and where to find the Postgres database attached to the app. These are parsed from the command line by Tornado’s options module and made available globally on the tornado.options.options variable. Continue reading “ipythonblocks.org Move: Part 4 – Application Updates”

ipythonblocks.org Move: Part 4 – Application Updates

ipythonblocks.org Move: Part 3 — Database Interface

This is Part 3 in a series of blog posts describing my move of ipythonblogs.org from Rackspace to Heroku. In this post I’ll describe the updates I’ve made to the database interface module of ipythonblocks.org. Other posts are:

  • Part 1: Introduction and Architecture
  • Part 2: Data Migration
  • Part 3: Database Interface Updates
  • Part 4: Application Updates

The big change to the database interface module was the switch from dataset to SQLAlchemy for database abstraction. This involves using the ORM models described in Part 2, removing the JSON de/serialization functions needed to use SQLite, removing use of memcached, and updating tests to use a Postgres database to match production. The full diff is here, but I’ll breakdown the important points below. Continue reading “ipythonblocks.org Move: Part 3 — Database Interface”

ipythonblocks.org Move: Part 3 — Database Interface

ipythonblocks.org Move: Part 2 – Data Migration

This is Part 2 in a series of blog posts describing my move of ipythonblogs.org from Rackspace to Heroku. In Part 1 of this series I described my motivation for the move and the broad changes I expect to make as part of the migration. In this post I’ll describe the grid data model and how I migrated the existing grid data from SQLite to Postgres. Other posts are:

  • Part 1: Introduction and Architecture
  • Part 2: Data Migration
  • Part 3: Database Interface Updates
  • Part 4: Application Updates

Continue reading “ipythonblocks.org Move: Part 2 – Data Migration”

ipythonblocks.org Move: Part 2 – Data Migration

ipythonblocks.org Move: Part 1

This is Part 1 in a series of blog posts describing my move of ipythonblogs.org from Rackspace to Heroku. In this first post I’ll describe the existing deployment and what I intend to change during the migration. Other posts are:

  • Part 1: Introduction and Architecture
  • Part 2: Data Migration
  • Part 3: Database Interface
  • Part 4: Application Updates

As a side project I maintain a Python library called ipythonblocks that displays colored grids in a Jupyter Notebook. (See also this intro blog post.) It can be useful for teaching or for a bit of fun art. I also maintain the website ipythonblocks.org that allows users to post their ipythonblocks grids on the internet to be shared. ipythonblocks.org has been hosted on Rackspace since I first launched it, but now I’m migrating the site to Heroku for easier maintenance and deployment. Continue reading “ipythonblocks.org Move: Part 1”

ipythonblocks.org Move: Part 1

Palettable 3.0 Released

I’m happy to announce the release of Palettable version 3.0. Palettable is a Python library that packages a variety of color palettes for use with matplotlib or really anywhere. Here’s the full diff since the last release.

This release includes a number of new palettes:

The new cmocean, matplotlib, and MyCarta palettes are created from data that contains 256 color points per palette. By default palettes are created with lengths 3-20 colors, but you can request longer ones via the get_map function. For example, to get the matplotlib Viridis palette with 200 color points:

In [2]: palettable.matplotlib.get_map('Viridis_200')

You can find Palettable on the web at:

P.S.: Here’s a little demo notebook.

Palettable 3.0 Released

Ally Work

As folks become more aware of the social justice challenges we face as a country and in our communities they often wonder how they can help. There’s not one really great answer because we all have different experience levels, means, axes of privilege, and available free time, but there are some common-sense “getting started” points. People with privilege who help social just causes are often called allies (but more on this word below), and this post is about how allies can learn how to help. A very short and incomplete TL;DR:

  • don’t center your own ego and feelings at the expense of others’
  • examine and change your own behavior so you don’t perpetuate bad habits
  • whole-heartedly listen to people with less privilege than you
  • educate yourself on the history and literature of the movements you are interested in
  • step up when you can

The following are several resources for people looking for specific things to think about and act on. This list is focused on diversity-in-tech, and especially gender diversity, so pointers to other resources would be great.

(A note about the word “ally”: there’s a real failing of language here because we use this noun to refer to privileged supporters of social justice causes, but the causes don’t need nouns, they need actions. You can’t do 10 supportive things and then have the label “ally” for life. It’s a process of doing what you can, when you can. This essay gives a great breakdown of the distinction: http://www.shakesville.com/2013/04/on-fixed-state-ally-model-vs-process.html.)

Note: This is based on a series of posts I’ve been putting together at work to educate coworkers on diversity and inclusion topics.

Ally Work

Dismissive Language

Dismissive language harks back to my post about microaggressions. It’s very easy to accidentally say something dismissive without realizing it. The most common examples of dismissive language I encounter are in descriptions of complex topics, often with the words “just” and “should”. For example, “All you need to do is just git rebase and everything should be fine” or “Just change this setting in your config and it should just work”. These constructions trivialize the listener’s problem and imply that they are stupid for not being able to figure it out on their own. (I’ve given technical examples here, but I imagine that this can happen in any context with complexity, like knitting or wet labs.)

I think it’s a good practice to try to completely remove the word “just” from your vocabulary, and watch your usage of the word “should” as well. At a fundamental level this is about putting yourself in someone else’s shoes, thinking about whether what you’re saying is appropriate given their level of experience, and making sure you’re not going to make them feel dumb by making something that’s actually hard/complex sound like you expect it to be obvious.

This is something we worry about a lot in Software Carpentry because we’re teaching complex computing topics to novice programmers. In fact, we recommend instructors completely avoid the word “just”. We discuss this a little in our instructor training materials:

Software Carpentry’s founder recently put together a short, 2.5 minute video giving examples of many of things you should not do when explaining complex topics to novices, such as using jargon and trivializing complexity. It’s worth a cringe-inducing watch:

The Recurse Center’s social rules also provide some great examples of things not to do: https://www.recurse.com/manual#sub-sec-social-rules

Note: This is based on a series of posts I’ve been putting together at work to educate coworkers on diversity and inclusion topics.

Dismissive Language

Unconscious Bias

This post came across my Twitter feed this week that talks about some of the spurious, meaningless reasons software engineering candidates are rejected. It’s challenging to evaluate candidates and we have a lot of productive discussions about how best to do that, but there some personal characteristics that we know are irrelevant to job performance: gender, race, age, sexual orientation, marital status, parental status, accent, where you grew up, etc. But even though we know those characteristics are irrelevant, they can still unconsciously affect how we evaluate and work with people through something called unconscious bias.

Unconscious bias manifests in some scary and surprising ways. For example, elementary school teachers give girls lower scores on math tests, unless they don’t know the gender of the students.

Another study examined gender bias in academic science. From the abstract: “In a randomized double-blind study (n = 127), science faculty from research- intensive universities rated the application materials of a student—who was randomly assigned either a male or female name—for a laboratory manager position. Faculty participants rated the male applicant as significantly more competent and hireable than the (identical) female applicant. These participants also selected a higher starting salary and offered more career mentoring to the male applicant. The gender of the faculty participants did not affect responses, such that female and male faculty were equally likely to exhibit bias against the female student.”

Bias can even have far reaching, unintended affects. Early marketing of personal computers targeted almost exclusively boys and that probably had an affect on the number of women seeking computer science degrees today.

One of the ways to combat unconscious bias is to learn more about it, which you can do with these resources:

If you’re evaluating a job candidate take a minute to think about how biases might be affecting your judgement. In fact, this can be a valuable thing to do in any interactions you have.

Note: This is based on a series of posts I’ve been putting together at work to educate coworkers on diversity and inclusion topics.

Unconscious Bias

Microaggressions

Microaggressions are small, often unintentional actions that can make others feel out of place. Examples can include asking where someone is from, acting surprised when someone doesn’t know something, or always expecting women to do office-keeping work. They can make people feel like they don’t belong and distract them from doing their actual jobs. “Micro”-aggressions might not sound so bad, but over time and many interactions they create an unwelcoming environment that can cause employees to leave or cause people to not want to attend events. As an example, a friend of mine almost swore off of all tech events after she went to a social event where everyone ignored her and talked to her lawyer husband instead.

It’s everyone’s job to make sure they’re actively creating a environment in which everyone feels that they fully belong. You can help by learning more about microaggressions and doing your best to avoid committing them (they hurt even when you don’t mean them to).

Here are a couple of good links that describe what migroaggressions are and the affect they have:

This post does a great job explaining how microagressions affect your life:

The Recurse Center’s social rules are very useful for helping you avoid committing microaggressions:

Note: This is based on a series of posts I’ve been putting together at work to educate coworkers on diversity and inclusion topics.

Microaggressions