Tools

Introducing Palettable

jiffyclub, , , , , Leave a comment

I wrote brewer2mpl a couple years ago to help people use colorbrewer2 color palettes in Python. Since then it’s expanded to include palettes from Tableau and the whimsical Wes Anderson Palettes Tumblr; and there’s plenty of room for more palettes from other sources. To encompass the growing scope, brewer2mpl has been renamed to Palettable! (Thanks to Paul Ivanov for the name.)

The Palettable API has also been updated for the IPython age. All available palettes are now loaded at import and are available for your tab-completion pleasure. Need the YlGnBu palette with nine colors? That’s now available at palettable.colorbrewer.sequential.YlGnBu_9. Reversed palettes are also available with a _r suffix.

I hope you find Palettable useful! You can find it on the web at:

P.S.: Here’s a little demo notebook.

Introducing Palettable

Snakeviz 0.2

jiffyclub5 Comments

It has been over two years since Erik Bray and I made the first release of SnakeViz 0.1, a tool for visualizing performance profiles of Python code. It had multiple performance bottlenecks, but it worked just well enough that it took me a long time to prioritize making improvements. That time has finally come around and I’m happy to announce that SnakeViz 0.2 is now available!

What’s New

The look and feel of SnakeViz remains much the same (see a screenshot), but there are some new things on the screen:

SnakeViz screenshot

  • Detailed function information when hovering over the visualization
  • Call stack list for tracking where you are when zooming the visualization
  • Control the depth of the displayed call tree
  • Limit the display of functions that take up relatively little time

Under the Hood

The first release of SnakeViz had some performance bottlenecks:

  • It tried to transfer a complete call tree from the server to the client as JSON
  • It tried to display the entire call tree in the sunburst visualization

Those limited the usefulness of SnakeViz with profiles that contained calls to a lot of functions. The version 0.2 release is an almost complete rewrite in order to make SnakeViz work with larger profiles.

The first limitation is addressed by moving the building of call trees into the client application. Profile data is passed from the server to the client in close to the same form as it’s available from Python’s pstats module. Once in the client, the profile data is used to construct call trees on demand for visualization.

The second limitation is addressed by limiting how much of the profile is visualized at once. Call trees are built only to a user specified depth and users can opt to omit functions that do not use much time from display. (The “depth” and “cutoff” controls.)

I and others have tested SnakeViz 0.2 with some fairly large profiles and found it works. You can read more about SnakeViz in the updated docs. Please give it a try! Issues can be reported on GitHub.

Snakeviz 0.2

State of Conda, Oct. 2014

jiffyclub, , , 6 Comments

I have been using Conda (via Miniconda) for managing my Python development environments and packages for close to a year now, so I thought I’d write up my thoughts so far for others.

Conda is both an environment manager (an alternative to virtualenv) and an installation tool (an alternative to pip). You can also use Conda to build your packages and distribute them via Binstar.

So, what does Conda do well and what needs improvement? Continue reading “State of Conda, Oct. 2014” →

State of Conda, Oct. 2014

If You Want to Build the NumPy and SciPy Docs

jiffyclubLeave a comment

This week docs.scipy.org has been down, but folks still need their NumPy and SciPy docs. To fill the gap until docs.scipy.org is back up I built the docs for only the latest stable releases and uploaded them to GitHub pages:

How to Build

(Note that I’m working on a Mac and these instructions are a little Mac/Linux oriented. The procedure on Windows would not be drastically different, though.)

Continue reading “If You Want to Build the NumPy and SciPy Docs” →

If You Want to Build the NumPy and SciPy Docs

Docker via Homebrew

jiffyclub, , 9 Comments

Docker is a great tool for getting lightweight, isolated Linux environments. It uses technology that doesn’t work natively on Macs. Until now you’ve had to boot into a VM to install and use Docker, but it’s now a little easier than that.

As of Docker 0.8 it can be run on Macs thanks to a specially developed, lightweight VirtualBox VM. There are official instructions for installing Docker on Mac, but with Homebrew and cask it’s even easier.

Follow the instructions on the cask homepage to install it. Cask is an extension to Homebrew for installing Mac binary packages via the command line. Think things like Chrome or Steam. Or VirtualBox. Running Docker on Mac requires VirtualBox so if you don’t have it already:

brew cask install virtualbox

Then install Docker and the helper tool boot2docker:

brew install docker
brew install boot2docker

boot2docker takes care of the VM that Docker runs in. To get things started it needs to download the Docker VM and start a daemon that the docker command line tool will talk to:

boot2docker init
boot2docker up

The docker command line tool should now be able to talk to the daemon and if you run docker version you should see a report for both a server and a client. (Note: When I ran boot2docker up it told me that the default port the daemon uses was already taken. I had to specify a different port via the DOCKER_HOST environment variable, which I now set in my shell configuration.)

If everything has gone well to this point you should now be able to start up a Docker instance. This command will drop you into a bash shell in Ubuntu:

docker run -i -t ubuntu /bin/bash

Use ctrl-D to exit. I find this especially helpful for very quickly getting to a Linux command line from my Mac for testing this or that, like checking what versions of software are installing by apt-get.

Visit the Docker documentation to learn more about what you can do with Docker and how to do it.

Docker via Homebrew

Using Conda Environments and the Fish Shell

jiffyclub, , , 4 Comments

I recently started over with a fresh development environment and decided to try something new: I’m using Python 3 via miniconda. The first real hiccup I’ve run into is that conda’s environment activation/deactivation scheme only works in bash or zsh. I use fish. There is an open PR to get fish support for conda but in the meantime I hacked something together to help me out.

"Activating" a conda environment does a couple of things:

  • Puts the environment’s "bin" directory at the front of the PATH environment variable.
  • Sets a CONDA_DEFAULT_ENV environment variable that tells conda in which environment to do things when none is specified.
  • Adds the environment name to the prompt ala virtualenv.

Deactivating the environment resets everything to its pre-activation state. The fish functions I put together work like this:

~ > type python
python is /Users/---/miniconda3/bin/python
~ > condactivate env-name
(env-name) ~ > type python
python is /Users/---/miniconda3/envs/env-name/bin/python
(env-name) ~ > deactivate
~ > type python
python is /Users/---/miniconda3/bin/python

Here’s the text of the functions:

function condalist -d 'List conda environments.'
for dir in (ls $HOME/miniconda3/envs)
echo $dir
end
end
function condactivate -d 'Activate a conda environment' -a cenv
if test -z $cenv
echo 'Usage: condactivate <env name>'
return 1
end
# condabin will be the path to the bin directory
# in the specified conda environment
set condabin $HOME/miniconda3/envs/$cenv/bin
# check whether the condabin directory actually exists and
# exit the function with an error status if it does not
if not test -d $condabin
echo 'Environment not found.'
return 1
end
# deactivate an existing conda environment if there is one
if set -q __CONDA_ENV_ACTIVE
deactivate
end
# save the current path
set -xg DEFAULT_PATH $PATH
# put the condabin directory at the front of the PATH
set -xg PATH $condabin $PATH
# this is an undocumented environmental variable that influences
# how conda behaves when you don't specify an environment for it.
# https://github.com/conda/conda/issues/473
set -xg CONDA_DEFAULT_ENV $cenv
# set up the prompt so it has the env name in it
functions -e __original_fish_prompt
functions -c fish_prompt __original_fish_prompt
function fish_prompt
set_color blue
echo -n '('$CONDA_DEFAULT_ENV') '
set_color normal
__original_fish_prompt
end
# flag for whether a conda environment has been set
set -xg __CONDA_ENV_ACTIVE 'true'
end
function deactivate -d 'Deactivate a conda environment'
if set -q __CONDA_ENV_ACTIVE
# set PATH back to its default before activating the conda env
set -xg PATH $DEFAULT_PATH
set -e DEFAULT_PATH
# unset this so that conda behaves according to its default behavior
set -e CONDA_DEFAULT_ENV
# reset to the original prompt
functions -e fish_prompt
functions -c __original_fish_prompt fish_prompt
functions -e __original_fish_prompt
set -e __CONDA_ENV_ACTIVE
end
end
# aliases so condactivate and deactivate can have shorter names
function ca -d 'Activate a conda environment'
condactivate $argv
end
function cda -d 'Deactivate a conda environment'
deactivate $argv
end
# complete conda environment names when activating
complete -c condactivate -xA -a "(condalist)"
complete -c ca -xA -a "(condalist)"
view raw conda.fish hosted with ❤ by GitHub

Or you can download it from https://gist.github.com/jiffyclub/9679788.

To use these, add them to the ~/.config/fish/ directory and source them from the end of the ~/.config/fish/config.fish file:

source $HOME/.config/fish/conda.fish
Using Conda Environments and the Fish Shell

Making Commits via the GitHub API

jiffyclub, , , , Leave a comment

For fun I’ve been learning a bit about the GitHub API. Using the API it’s possible to do just about everything you can do on GitHub itself, from commenting on PRs to adding commits to a repo. Here I’m going to show how to do add commits to a repo on GitHub. A notebook demonstrating things with code is available here, but you may want to read this post first for the high level view.

Choosing a Client Library

The GitHub API is an HTTP interface so you can talk to it via any tool that speaks HTTP, including things like curl. To make programming with the API simpler there are a number of libraries that allow communicate with GitHub via means native to whatever language you’re using. I’m using Python and I went with the github3.py library based on its Python 3 compatibility, active development, and good documentation.

Making Commits

The repository api is the gateway for doing anything to a repo. In github3.py this is corresponds to the repository module.

Modifying a Single File

The special case of making a commit affecting a single file is much simpler than affecting multiple files. Creating, updating, and deleting a file can be done via a single API call once you have enough information to specify what you want done.

Modifying Multiple Files

Making a commit affecting multiple files requires making multiple API calls and some understanding of Git’s internal data store. That’s because to change multiple files you have to add all the changes to the repo one at a time before making a commit. The process is outlined in full in the API docs about Git data.

I should note that I think deleting multiple files in a single commit requires a slightly different procedure, one I’ll cover in another post.

That’s the overview, look over the notebook for the code! http://nbviewer.ipython.org/gist/jiffyclub/9235955

Making Commits via the GitHub API

Writing WordPress Posts in Markdown

jiffyclub, , 3 Comments

Pen and Pants is hosted by WordPress, but I write my blog posts in my favorite text editor using Markdown. That way I have all the conveniences those afford and I can archive the posts in plain text on GitHub.

The tricky part is going from the .md files to some text I can paste into the input box in WordPress. I learned today that you can write posts in Markdown, but that still doesn’t work perfectly for me because WordPress treats new lines within blocks as hard breaks. (When writing posts I break all lines before 80 characters for more convenient editing and diffing. Keeping all those breaks literal doesn’t translate well to web pages.)

Today, thanks to Ethan White, I figured out that Pandoc can help. By converting my Markdown to Markdown with the --no-wrap flag Pandoc will output paragraphs on a single line but otherwise give me regular Markdown. The command I use looks like this:

pandoc -f markdown -t markdown --no-wrap blog-post.md

I can take the output of that and past it into WordPress’ text input box (after ticking the box to allow Markdown when writing posts).

Note that if you use fenced codeblocks (as on GitHub) WordPress will convert that into its special source code widget. If instead you want something presented using only <pre><code> tags then use indentation to indicate it is pre-formatted text.

Tips for Mac Users

If you use Homebrew you can install Pandoc via the cask add on:

brew cask install pandoc

To copy the output of pandoc straight to the clipboard you can use the pbcopy command:

pandoc -f markdown -t markdown --no-wrap blog-post.md | pbcopy
Writing WordPress Posts in Markdown