VisiData Blog Feed

Estimates of current and future usage

2021-10-29T00:00:00Z

Estimates of current and future usage

[Since v1.3, the motd feature has served as a usage counter. This upsets some people even though it can be disabled. Here's an explanation of why this is on by default.]

Our usage data gives us some good information about the open source contributor funnel. As of October 2021, it seems to break down this way:

~3000 monthly users
~300 daily users
~30 community members
~3 regular contributors

Usage Growth

Here's what VisiData usage looked like for the first 2.5 years of its existence: from 40 daily M-F users in early 2018, to almost 100 in mid-2019.

Here's what the usage graph looks like now:

This graph above has 500 daily users on the Y axis. Let's extrapolate another 4x to 2000 and another 5 years:

This graph makes even a "modest" 4x usage growth seem unlikely. But consider this next graph, which is the view in 2019 (2.5 years old and 2.5 years ago), until today), with the axes expanded to what they are today:

Its growth to today would have seemed as unlikely then, and yet here we are.

What will VisiData usage look like in 5 years? If it continues to double every year, then in 5 years that would be 32x; possibly exceeding 10,000 daily users! That seems a little excessive. But with millions of terminal users and millions of Python users and millions of data professionals who could benefit from a quick dive into a new dataset and whip it into shape, it's not impossible.

VisiData Turns 5

2021-10-29T00:00:00Z

Today, we celebrate 5 years of VisiData. In honor of the occasion, I thought I'd put together a little retrospective, and some thoughts on what the future might hold.

What is VisiData?

VisiData is an interactive tabulator. Like VisiCalc is an interactive calculator, and vim is an interactive text editor. It's the ultimate tool for dealing with tabular data: exploring, cleaning, analyzing, and transmuting. VisiData is not a spreadsheet, where cells are the first-class elements. However, it can load and interact with data in spreadsheet format, and many other formats too, even some things you might not think of at first: like filesystem metadata and API results and packet captures. Because everything is data.

History

I've been working on VisiData since October 2016, and in the beginning, there was only one user (me).

The first "release" was v0.14 to r/Python in November 2016 but no one noticed. Then in December I shared v0.37 to r/pystats, which got a bit more attention, but was scorned for not having any tests.

Back then it was a single 1000-line script that had loaders for csv, json, xlsx, hdf5, and could already create frequency tables and join sheets. Many people would be quite happy with just this functionality.

I kept going.

2017: pre-v1

After only a few months, VisiData was already incredibly useful. But I had a lot more ideas, so in 2017 I worked on VisiData extensively, always thinking a 1.0 was just around the corner. The first major release prior to v1.0 was in June 2017, and made it to the top of Hacker News.

I didn't have firm criteria for "version 1.0" was, I just knew I didn't want it to be a version 0 project forever. I mainly wanted 1.0 to be robust and useful without major maintainance.

2018: v1 era

In Jan 2018, I released version 1.0, and then took a step back. Shortly after 1.0 I started getting some basic usage data. I kept plugging away at it, and Anja (who I met at the Recurse Center in 2017) became involved in the project. Jeremy Singer-Vine, who had discovered VisiData the year before, wrote the excellent Introduction to VisiData tutorial.

2019: v1.5.2 era

At the end of 2018 I took a full-time job, so in January of 2019 we released a "final" v1.5.2 which was very reliable, and it was the last official release for almost 2 years.

Here's what VisiData usage looked like for the first 2.5 years of its existence: from 40 daily M-F users in early 2018, to almost 100 in mid-2019.

During 2019 I worked full-time, but still added functionality to VisiData on the side, or as I needed features for my paid work. These stayed in the develop branch, and only used by a handful of dedicated users.

2020: pre-v2 era

I wanted to clean up some elements of the API but I was aware of the responsibilities of maintaining a stable platform. So I let the develop branch accumulate changes, and we only made some internal beta releases.

Then 2020 happened, and I stopped working full-time to keep things together. Anja and I put a bunch of effort into cleaning up the API, and we released v2.0 in October 2020.

2021: v2 era

In 2021, we've continued to put out new releases every few months. The Devottys and I released the xdplayer terminal crossword player and DarkDraw (a Unicode drawing tool). I reorganized most of my VisiData addons into VisiData Plus.

Here's what the usage graph looks like now:

This usage data gives us some good information about the open source contributor funnel. In VisiData's case, it breaks down this way:

~3000 monthly users
~300 daily users
~30 community members
~3 regular contributors

tl;dr: the present

In short, VisiData now has some ~15k lines of code, some thousands of regular users, and is frequently mentioned wherever people talk about tabular data tools and/or the terminal. Its architecture for working with row/column data is powerful, and the interface is remarkably flexible and ergonomic for experienced users. Users regularly call VisiData "delightful", "amazing", and say they "love" it. My favorite response (as for most of my work) is "holy shit".

The Future of VisiData

According to Lindy's Law, the life expectancy of software is proportional to its age. So now that VisiData has been around for 5 years, we can look forward to and plan for another 5 years.

Usage Growth

The second graph has 4x daily users on the Y axis. Let's extrapolate this graph another 4x and another 5 years:

Its growth to today would have seemed as unlikely then, and yet here we are.

More Usability and Teachability

Like many powertools, VisiData has so much functionality that it can be offputting to new users. Even though a handful of commands would already provide a large amount of power and flexibility, new users often bounce off of it because it just seems overwhelming.

So a major and ongoing effort, is to make VisiData even easier to use, while not degrading the experience for power users, and in fact guiding new users into becoming power users without a lot of effort. The new menu system in v2.6 is a big step towards better discoverability. Next on the list will be a sidebar for "wizard" commands like join, group, and plot; a more holistic and guided fancy chooser.

Bigger Data

VisiData is great for dealing with a few million rows of data. But more than that, and it starts to buckle. Modern hardware and databases can readily work with 100 million rows on a single machine if they are stored and accessed correctly. In the next few years, VisiData will be able to access these larger datasets without having to load everything into memory. Imagine being able to browse a BigQuery table using Shift+F for a Frequency Sheet and Enter to dive into a subset, and then copying the resulting SQL into a notebook!

Bonkers Plugins

Like Emacs is to text, VisiData is to tables. If my wildest ambitions were fulfilled, VisiData would be the solution in the following analogy:

Emacs : text : Lisp :: VisiData : tables : Python

and with that in mind, there's lots of room to grow!

Absurd Amounts of Polish

As always, what makes VisiData so great to use, is the absurd amount of polish and attention to detail. One by one, we find the frictions and try to remove them. The minor inconsistencies. The edge cases. The little things that are deemed "not worth it" in most software.

Here's to another 5 years of VisiData!

Thanks again to our current corporate sponsors October Swimmer and General UI, and to all of the people who have contributed to make VisiData awesome!

VisiData.org Redesign

2021-08-13T00:00:00Z

Hello! My name is Luke Harris, and I'm a web designer. I recently completed the new site design for VisiData.org, and wanted to outline the goals and challenges. Let's dive in.

Goals for the design

Effectively communicate new releases and blog posts
Better usability - consistent navigation and structure
Easier discovery of documentation and features
Searchable documentation with an easy-to-browse sidebar layout
Statically generated from existing Markdown files, with templates as needed
Easy to update and maintain by the VisiData creator and contributors
RSS feeds for blog posts and releases

Colophon

This website was originally built with a custom-made static site generator (SSG) and is a mix of regular Markdown files and reStructuredText parsed by pandoc.

This site is now statically generated with 11ty, and uses TailwindCSS for styling.

Challenges overcome

reStructured Text

The docs were written in a mix of reStructuredText (RST) and Markdown, which proved a challenge as 11ty does not support RST. 11ty does support HTML files, so Saul wrote a script to run the docs through pandoc and output the resulting HTML files into a directory for 11ty.

Documentation search

We needed a quick self-hosted solution to search the documentation, and rely on as few external dependencies as possible. This ruled out popular search engine Algolia, and led me to lunr.js, which can be fed a JSON file (generated by 11ty) of the index, and then run the search client-side in the browser. If JavaScript isn't supported in the browser, the search results page falls back to a DuckDuckGo site search link.

Updatable components

The site features 3 JSON files for easy updating of sections of the site that see frequent updates, including the sponsors in the footer area, the header navigation, and the testimonials.

Design highlights

Command demo

This demo existed prior to the redesign on the old site. I gave it a facelift and designed the terminal output area to use a monospaced font. If you select the text, the $ is left out, allowing for easy copy-and-paste of commands.

Features and Getting started

Using CSS Grid, I designed a highlighted features area with one block spanning two columns, making for a neat effect.

Releases and Blog lists on homepage

It's important to quickly be able to catch up on what's new with an open-source project. Adding the blog posts and releases list to the homepage was easily accomplished with 11ty's post collections.

Before and after

Before:

After:

Issues

If you come across any bugs or issues with the new design, let us know by opening an issue on the repo.

Originally posted on luke.work

10 Ways to Use VisiData

2021-07-24T00:00:00Z

1. Explore datasets effortlessly, no matter the format

$ vd albums.json dropbox/*.xls* agents.sqlite

Use the arrow keys to move around and q to quit.

VisiData loads data from many different sources and formats, including anything pandas can load!

New loaders are included in every release, and you can create one yourself easily.

2. Convert from any input format into any output format

$ vd -b input.csv -o output.json

3. Interactively select replacement for cli tools

$ mv $(ls | vd) /trash

Select files using any of the selection commands.

Press " to pull them into their own sheet.

Press Ctrl+Q to exit VisiData and write the current sheet to stdout.

4. Scrape HTML table data from a webpage

$ vd https://en.wikipedia.org/wiki/List_of_largest_cities

5. Parse and explore text-based log files

$ vd output.log

Use : to split each line by some string.

Use ; with a regex with capture groups like (\d+).

6. Explore a Python object from the REPL

>>> visidata.pyobj.view(obj)

7. Explore any pandas DataFrame

>>> visidata.vd.view_pandas(df)

8. Create an adhoc data pipeline

$ vd -b -p foo.vd

VisiData can also be used in batch mode with scripts without requiring any interaction.

Save the cmdlog to a .vd file with Ctrl+D foo.vd, then replay the saved .vd file:

9. rename a bunch of files

$ vd <directory>

Edit the filenames (documentation on editing commands). Press z Ctrl+S to commit the changes (before v2.0, use Ctrl+S).

Watch this video showing how to use the directory browser.

10. see all the available options and commands

There are many other ways to use VisiData! To learn more, press Ctrl+H to see the manpage, or z Ctrl+H to see the full list of commands available on the current sheet.

[written by Saul Pwanson 2019-03-24]
[updated on 2020-10-15]
[updated on 2021-07-24]
[updated on 2022-03-21]

New feature: Persistent input history

2021-01-10T00:00:00Z

Like most tools, VisiData keeps a history of your responses to its prompts, so you can scroll through them in subsequent prompts using the Up and Down arrows.

Up until now, these histories would only be available until the end of the VisiData session. By default this is still the case, but if you set options.input_history to any value, the history will persist between sessions.

options.input_history = 'lastinputs'

This will put the history in .visidata/lastinputs.jsonl.

Input histories are separated based on the type of the input (e.g., Python expressions, regex search, regex split, filenames), The Last Inputs Sheet can be opened with the open-inputs command, and edited/saved if desired.

Many thanks to AJ Kerrigan who subbed to Saul at the Workflow Optimist level on Patreon, and requested that this feature be kept open in our issues list. This feature is currently in the develop branch and slated for release in v2.2.

[written by Anja Boskovic 2021-01-10] [edited by Saul Pwanson 2021-01-10]

Expansion of CLI Options

2020-07-07T00:00:00Z

Hello VisiDaters!

Recently, Saul has pushed a re-work of the CLI options to the develop branch. I am particularly excited about this change, and so I wanted to cover it in detail.

I am going to be referencing two options; -f, alias for --filetype, and -d, alias for --delimiter.

By default, VisiData deduces the filetype of a source from its extension. -f allows you to manually declare the filetype, overriding deduction by extension. -d sets the delimiter for TSV files.

The Additions

vd --help will now open the manpage
VisiData now supports the setting of sheet-specific options, as well as global, through the CLI.
vd -f json foo.json.gz now applies to the inner filetype.

Historically, if you did,

vd -f csv foo.tsv bar.csv

the option -f would then get applied as a global option to every source. There was no way to communicate to VisiData distinct filetypes for each source.

With the new additions, you can do this,

vd -f tsv foo.tsv -f csv bar.csv

Now, -f tsv would be applied to foo.tsv, and -f csv would be applied to bar.csv.

How Sheet-Specific CLI Options Work

--options will apply as sheet-specific options to the sources which follow them.
The final setting for a given option will be considered the CLI-given override setting. It will be applied to all sources loaded by the CLI, if that source does not have its own sheet-specific setting. This means that if an option is only set once, it will be applied to all of the sources.
If an option is preceded by -g it will be set globally for all sheets in the session, just like in the previous versions of VisiData. Some special options, like -p (replay), are always going to be global, and will not require the -g prefix.

Examples:

To specify a different delimiter for each file,

vd -d '|' pipes.tsv -d ',' commas.tsv

To set a global delimiter,

vd pipes.tsv commas.tsv -g -d '|'
vd -g -d '|' pipes.tsv commas.tsv

To specify a filetype for all of the files provided through the CLI,

vd foo bar -f tsv

Limitations

there is no way to make an option apply only to the immediately following source
nor a way to unset an option after setting it

As always, please let us know how you feel about it! Share a tweet, or stop by for a chat in freenode #visidata.

[written by Anja Boskovic 2020-07-07] [updated by Anja Boskovic 2020-07-26]

VisiData's Limits on Growth

2020-07-01T00:00:00Z

I added a page on the website, /limits, summarising a list of things we wish were different, but would require unwanted adjustments to the architecture to change.

Something I learned from spending time with Saul is that building software always requires making decisions around trade-offs. Sometimes, the decision about the trade-off really should lie with the user, which is part of the reason we added so many options.

Sometimes, though, to fix it would require a large architectural overhaul, or would involve a trade-off that is just..,Worse in every way. Collecting these stories is what the limits page is for.

The implications of tabular data architecture

2020-06-18T00:00:00Z

A known bug in VisiData (#413) is that the contents of SettableColumns created with addcol-* are not able to be yanked (copied to VisiData's internal clipboard).

The reason behind why this happens illuminates something significant behind VisiData's architecture.

There are two main ways table data can be organized: row-major and column-major. Row-major keeps each row together, whereas column-major keeps each column together. SQL databases are an example of a row-major data analysis tool (if they support row-level locking). Python pandas is an example of a column-major library.

Users might intuitively feel like VisiData is column-major, since it has so many column-centric commands, but architecturally it's actually row-major. This is so we can use objects as rows, and also have heterogenous column values. You can toss around bulk row references like they're nothing. Operations like load and filter are really fast.

SettableColumn (and other columns which store data inside themselves "about" rows) is column-major, and so the distinctions are blurred. It does happen to work pretty well for most things, but it's trepidacious and kind of off-label.

This is the fundamental issue behind #413 and why it can't be fixed.

This is also why working with the pandas loader (or any columnar store like raw parquet files) can be tricky. You have the hammering of row-oriented semantics around a lower level functionality or storage layer that is fundamentally column-oriented. It's awkward and carries a stiff performance penalty. VisiData is designed for loading and filtering over aggregating and saving.

In the future, perhaps pandas should be a completely different kind of Sheet. A TableSheet would actually be the row-oriented sheet, and there's another ColumnSheet that's column-oriented. Operations would be implemented differently, and maybe different ones would be available.

However, in the meanwhile, with this awareness we can make cleaner choices within VisiData, and understand other implications this can have for different loaders and commands.

[Written by Saul Pwanson 2020-06-17] [Edited by Anja Boskovic 2020-06-17]

'VisiData is like duct tape for your data', an interview with Saul

2020-05-16T00:00:00Z

Saul was recently interviewed on The Changelog podcast!

The Changelog 394: VisiData is like duct tape for your data

It also has a written transcript available.

This is a very wide-ranging interview. The first quarter covers his discovery of an egregious case of crossword plagiarism. If you want more, you can additionally check out How A File Format Led To a Crossword Scandal, the talk he gave at csv,conf in 2019.

Next, he talked about our collaboration in creating the biograph framework for our own personal histomaps.

In the second half, he talks about the side-effects of offering VisiData as a gift to the world. Open-sourcing software involves much more than just posting your code to a GitHub repository. There are insecurities, cultural practices, and responsibilities that come with it.

I hope you enjoy the interview! I really did.

Unloaded Sheets

2020-02-11T00:00:00Z

If after you pass multiple files to vd on the command-line, you go straight to the Sheets Sheet, you'll see that only the first one is loaded:

When a sheet is first created, it is considered "unloaded" (literally sheet.rows is UNLOADED). It can stay in this dormant state indefinitely, and is quite lightweight.

Then, as it's about to be drawn for the first time, the sheet starts its loading process (via sheet.reload()), to read the data into memory.

These reloads are spawned in a separate thread; this is what makes VisiData responsive as a sheet is loading.

But there is a problem here. If too many large sheets are loading at once, your laptop supercomputer would still slow to a crawl.

So if you have 100GB of .csv files in a directory and you do vd *.csv (which is a reasonable thing to want to do, especially if you have 100GB of .csv), VisiData only starts loading the top sheet (the first one given on the command-line) in order to draw it. The other sheets remain unloaded, as you can see on the SheetsSheet. When you jump to one of the unloaded sheets, it will start loading.

Now, if you have a bunch of small sheets and go to the Sheets Sheet expecting to compare or join them straight away, you might be disappointed to find them unloaded and unusable.

Fear not! To load multiple sheets at the same time, go to the Sheets Sheet, select the sheets you want to load, and then reload-selected with g Ctrl+R.

Now you can go forth and jump around bundles of csvs to your heart's delight, secure in the knowledge that VisiData will only read into memory what you intentionally want to engage with.

[written by Saul Pwanson 2020-02-10]

[edited by Anja Kefala 2020-02-11]

New Features in 2.-3: repeat-last and repeat-input

2020-02-09T00:00:00Z

Onto the develop branch, we have pushed two new commands: repeat-last and repeat-input.

These commands had been originally requested on the #visidata IRC channel. The design spec was then transcribed onto issue #441.

The aim was to have something spiritually similar vim's . (dot) command.

repeat-input runs the previous command, along with any previous input to that command. repeat-last runs the most recent command with an empty, queried input.

For the moment, they are unbound. To use any command with its longname, press SPACE followed by the longname. For example: SPACE repeat-last.

To bind the commands to your preferential keystroke (the following example uses .), add to your ~/.visidatarc`:

Sheet.bindkey('.', 'repeat-input')
Sheet.bindkey('z.', 'repeat-last')

Or to take advantage of a currently unused keybinding:

Sheet.bindkey(ALT + 'r', 'repeat-input')
Sheet.bindkey('z' + ALT + 'r', 'repeat-last')

As always, the features on develop are experimental. If you do use them, please let us know how you feel about them! Post an issue on Github, share a tweet, or stop by for a chat in freenode #visidata.

[written by Anja Boskovic 2020-02-09]

New feature in 2.-3: Split Window

2020-02-02T00:00:00Z

Hey VisiDaters, check it out!

In the bleeding-edge develop branch, VisiData can now show TWO sheets at the same time:

The ability to see two sheets on the screen at the same time has been on the v2.0 roadmap for a long time. I've even caught myself lamenting a few times that, if we had split-screen, the design for some feature would be straightforward. Deep down I also think it might open up a whole new dimension of user interface, where movement on the active window instantly updates the inactive window.

So I've given it a few tries already; it's not that technically challenging to do, but the interface for it remained elusive After talking with @tsibley and @anjakefala and sitting with it for long enough, last week I finally put together a working prototype. It's serviceable but not perfect, so hopefully it can serve as a launching pad for discussion about the finer points of split-screen workflow.

`options.disp_splitwin_pct`

The main option is disp_splitwin_pct, which specifies the height of the top window as a percent of the full screen height. Currently, a positive number puts the top sheet in the top window, whereas a negative number keeps the same layout, but puts the top sheet in the bottom window.

Commands

Shift+Z splits the screen in half so that the second-most-recently accessed sheet is visible
g Shift+Z sets the active sheet to the whole screen (effectively disable splitwin)
z Shift+Z specifies options.disp_splitwin_pct exactly
Ctrl+^ (which already jumps to the previous sheet) naturally swaps the layout without changing the active panel
TAB swaps the active panel, without changing the layout. (note that this means the replay-advance command is currently disabled; I'm still not sure how all this is going to work.)

Its full implementation could solve several issues (choosing jointype/aggregators, popup command help) and would make a two panel orthodox file manager all but inevitable.

Theme options

color_top_status is applied to the "top window status bar" -- which is in the middle of the screen. Set by default to underline to divide the screen visually.
color_inactive_status is applied to the status bar of the inactive window.
color_active_status is applied to the status bar of the active window.

Limitations

only 2 sheets at the same time; from one to two sheets is a quantum leap by itself and a reasonable constraint (at least to start with).
only top/bottom split is supported; this implementation could be extended to include options.disp_splitwin which might specify e.g. top or left to indicate the panel of the top sheet.
it can't be specified as a CLI option (no disp_ or color_ option can).
canvas/graph sheets aren't reset and may have errors that require Ctrl+L to resize properly.

This is still quite experimental so if you use this feature, let me know how you feel about it! Post an issue on Github, share a tweet , or stop by for a chat in freenode #visidata.

[written by Saul Pwanson 2020-02-02]

Hello World!

2020-01-19T00:00:00Z

I have always wanted a home for longform conversations on VisiData.

For the past three years, we have experimented with tinyletters, videos, release notes, podcast interviews, patreon posts, twitter threads, and responses to github issues.

I now want to consolidate those conversations into a single place. The backfill of this blog hopes to accomplish this. Additionally, I wanted a space for writing that is more impulsive than a newsletter, and merits more room for thought than a tweet. I am hoping this blog will become that space.

Thanks for sticking around for the past 3 years. =) Welcome to the next stage in this experiment.

VisiData Case Study #1 Transcript: summing donation data from a webpage

2020-01-15T00:00:00Z

This case study was originally a video, that was published on 2018-04-01. The video was designed and recorded by Saul Pwanson with help from Tara Hoch.

It was transcribed by E on 2019-12-11 and then updated and converted into a blog post by Anja Boskovic.

This walkthrough will use VisiData to take a paragraph of donation information from a website (beginners.re), and then get it into a form that can be easily summed.

In the process, the following commands/features will be covered:

adding custom functions to ~/.visidatarc
piping stdin into VisiData
Ctrl+O in any edit field to open it with a custom external editor (specified with $EDITOR)
: to split columns
Shift+M to melt (un-pivot)
; to match by regex capture group
$ to type columns as "currency" (a dirty float)
z+ to apply an aggregator statistic to the current column and show the result
~ to type a column as a string
= to create a new column by Python expression

Since the donations are all in different currencies, Saul made a plugin called USD. He uses the plugin's USD() function at the final step in the case study. USD() takes a number and a currency code or currency symbol, and converts it into US dollars based on the current day's currency rates. If you want to follow along with the final step of the tutorial, before you launch your VisiData session you will need to add the USD() code to your ~/.visidatarc or install it as a plugin. Then its functions will be added to your session's scope. This is completely optional, and only comes to play at the end.

Since the video's creation, fixer.io has also added a user api key requirement. To take advantage of the usd plugin, you will also need to create an account, grab an api key, and then set options.fixer_key in your .visidatarc in your $HOME directory.

`.visidatarc`

options.fixer_key = 'insert_here'

Transcription of case study starts

Saul: I came across a book called "Reverse Engineering for Beginners," written by Dennis Yurichev. It seems like it's a pretty good book; he's got a bunch information about it, and he also lists the donors who have supported him. I actually really appreciate his transparency here, that he will share all this information on his website. It made me curious as to how much money had been donated to him overall.

So you'll notice that this information here is in a single line. It's actually not that bad, but it's not in a form that we could easily do something with. So we're going to use VisiData to pick it apart, and try to do a sum of how much money was donated to him.

First we are going to cut and paste this text into our terminal window, and pipe into VisiData directly.

echo '2 * Oleg Vygovsky (50+100 UAH), Daniel Bilar ($50), James Truscott ($4.5), Luis Rocha ($63), Joris van de Vis ($127), Richard S Shultz ($20), Jang Minchang ($20), Shade Atlas (5 AUD), Yao Xiao ($10), Pawel Szczur (40 CHF), Justin Simms ($20), Shawn the R0ck ($27), Ki Chan Ahn ($50), Triop AB (100 SEK), Ange Albertini (€10+50), Sergey Lukianov (300 RUR), Ludvig Gislason (200 SEK), Gérard Labadie (€40), Sergey Volchkov (10 AUD), Vankayala Vigneswararao ($50), Philippe Teuwen ($4), Martin Haeberli ($10), Victor Cazacov (€5), Tobias Sturzenegger (10 CHF), Sonny Thai ($15), Bayna AlZaabi ($75), Redfive B.V. (€25), Joona Oskari Heikkilä (€5), Marshall Bishop ($50), Nicolas Werner (€12), Jeremy Brown ($100), Alexandre Borges ($25), Vladimir Dikovski (€50), Jiarui Hong (100.00 SEK), Jim Di (500 RUR), Tan Vincent ($30), Sri Harsha Kandrakota (10 AUD), Pillay Harish (10 SGD), Timur Valiev (230 RUR), Carlos Garcia Prado (€10), Salikov Alexander (500 RUR), Oliver Whitehouse (30 GBP), Katy Moe ($14), Maxim Dyakonov ($3), Sebastian Aguilera (€20), Hans-Martin Münch (€15), Jarle Thorsen (100 NOK), Vitaly Osipov ($100), Yuri Romanov (1000 RUR), Aliaksandr Autayeu (€10), Tudor Azoitei ($40), Z0vsky (€10), Yu Dai ($10), Anonymous ($15), Vladislav Chelnokov ($25), Nenad Noveljic ($50), Ryan Smith ($25), Andreas Schommer (€5). ' | vd

Now VisiData has that as one of its rows.

If we reduce this column to just a few characters here, you can see it's actually quite a long field. If you wanted to view the field and see all the contents of it, you can edit it (with e), and you can scroll across the whole thing.

You can also press Ctrl+O to open any input line in your own editor: in this case it's vim, in the terminal. With this, you can see the entire contents. You can also make edits and save them off; they'll be reflected back in the cell. For now, we are just going to quit out of this and cancel the edit.

What we want to do, is split up that single line according to the individual donations. Each donation is separated by a comma, so we're going to use the split command (:). The split command allows you to split the text in any column into two or more columns, based on a provided regex. In this case, the split regex is just a comma (,). That will create columns for every individual donation.

Now, we're going to hide this first column (-), and now we've got every donation in its own cell, which is a great start. This is a horizontal form of the data, but we want the data to be in a vertical form, because that's what VisiData expects. VisiData does statistical operations within columns.

So we're going to use a command in VisiData called 'melt': it's like an un-pivot. It takes data from a horizontal form and swings it down into a vertical form. Now you can see the column names on the left, and the cell values on the right. Now we're ready to go.

We can move our cursor to the Value column, and we're going to use another command. The ; key extracts part of a column according to regex. We're going to pull out the donation amount from within the parentheses, by constructing a regex. This is the standard regex for extract everything from inside the parentheses: $(.*)$. The inner set of parentheses (.*) is the capture group. Everything included in the capture group is going to be pulled into a new column.

If we wanted to, we could just sum up this set of numbers. It wouldn't be very meaningful, because they're all in different currency units, but we could do it.

There is a type, assigned to keystroke $, which I call a currency type. It's really just a dirty float: it takes the initial non-numeric characters and converst them into a float. The trailing non-numeric characters are stripped off. The cells that have a symbol in the middle can't converted, so they will show up as errors. We will ignore those for now. And so now we can approach this as just a column of numbers.

+ is used to put an aggregator on a column, which then shows up in future sheets. Here, we're going to use the z+ command, which applies an aggregator to the current column and shows the result immediately. And so if we want to sum up the current column, we type z+ sum and there we go.

This is about 4,300 units of currency. Again, it's not very meaningful, because they're all in different currencies. Since they are in different currencies, here is where the USD plugin comes into play. It takes a number and a currency code or currency symbol, and converts it into US dollars based on the current day's currency rates.

To use it, let's convert our currencies back to strings. This will involve using the ~ command. It was so-chosen because it looks like a string to me. Pressing it will type the current column back to a string, like it originally was.

Now, we are going to use = to create a new column based on an input Python expression. With that, we are going to use the USD function on this Value_re0 column.

I'm then going to convert this USD output into a float type. Now we can see that, for instance, this €50 is actually ~$55 in US money. Others that were already in US money just remain the same.

Now, we can do the same z+ sum as we did before, and see the value in US dollars as of today's rates. That is about $1,431 worth of value: that's the total donations he got from people for his book. It's not quite the full total, because we're missing the ones with errors and such. We could do some more things to make that better, but this is a rough general calculation.

Thanks very much for using VisiData, and I will see you next time!

Exploratory Data Analysis Made Easy At The Command Line - Episode 230

2019-10-30T00:00:00Z

This is an edited transcript of an interview about VisiData in this episode of Python.__init__. It was carefully transcribed and edited by E (@cel10e on twitter) on 2019-10-30.

Enjoy!

Tobias' intro [00:00]

[Intro music plays.] Tobias Macey: Hello, and welcome to Podcast.init, the podcast about Python and the people who make it great.

When you're ready to launch your next app or want to try a project you hear about on the show, you'll need somewhere to deploy it. So take a look at our friends over at LINODE. With 200 gigabit private networking, scalable shared block storage, node balancers, and a 40 gigabit public network, all controlled by a brand new API, you get everything you need to scale up. And for your tasks that need fast computation, such as training machine learning models and running your continuous integration, they just launched dedicated CPU instances. Go to pythonpodcast.com/linode — that's L-I-N-O-D-E — today to get a $20 credit and launch a new server in under a minute. And don't forget to thank them for their continued support of this show.

And you listen to this show to learn and stay up to date with the ways that Python is being used, including the latest in machine learning and data analysis. For even more opportunities to meet, listen, and learn from your peers, you don't want to miss out on this year's conference season. We have partnered with organizations such as O'Reilly Media, Dataversity, Corinium Global Intelligence, and Data Council. Upcoming events include the O'Reilly AI Conference, the Strata Data Conference, the combined events of the Data Architecture Summit and Graph Forum and Data Council in Barcelona. Go to pythonpodcast.com/conferences today to learn more about these and other events, and take advantage of our partner discounts when you register.

Saul intro [01:37]

Tobias Macey: Your host as usual is Tobias Macey, and today I'm interviewing Saul Pwanson about VisiData, a terminal-oriented interactive multi-tool for tabular data. So, Saul, can you start by introducing yourself?

Saul Pwanson: Yeah, my name is Saul, I've been working in the software industry for a while, and I'm in Seattle.

Tobias Macey: And do you remember how you first got introduced to Python?

Saul Pwanson: Yeah, it was for work back in 2004. I was at a startup which was using it for their apps, and I picked it up there, because that's just what you do, right?

Tobias Macey: And, so, at some point, you decided that you needed to start building your own multi-tool for working with data, particularly in tabular formats, and I'm wondering if you could just start by describing a bit about the tool that you built in the form of VisiData, and some of the backstory of how it got started?

Saul Pwanson: Yeah. I was working at a company called F5 Networks in 2011, I think, and I built up an early prototype of VisiData for them. I didn't know it would become VisiData at the time, it was just a configuration and exploration tool for their own networking hardware. But, as I got used to using it, it was fun, I found that the concept was very flexible, and I kept adding more and more stuff to it.

And then after I left F5, I found myself missing it. I kept wanting to use it to look at HDF5 files or other tabular data that I had. And so, when I was at another company in 2016, and that company was winding down, I realized that I was turning 40, and looking back over my career of almost 20 years, I didn't have very much to show for it. And I was like, "well, I've been wanting this tool, so let me just start that again." I couldn't use the code that I had written at F5, obviously. So I was like, "if I'm gonna rewrite it, let me do it again from scratch, and I'll make it right once and for all as the tool that I want to use." And so that was the genesis of VisiData.

Tobias Macey: There are a couple interesting things in there. One is the fact that you had to try and rewrite this tool from scratch, largely from memory, without copying the exact code that you had written. And then also, you mentioned HDF5 files, and I'm curious: have you experimented with PyTables at all, or any of the other libraries for that particular data format, and what was it that you found lacking that you wanted to have present in VisiData?

Saul Pwanson: So I hadn't used PyTables, but I've used a lot of other Python tools for using HDF5 files. And just like with any complicated format, you can do anything you want to with Python, it's really great for that. But it's still kind of a hassle – you still have to write code to do it. And often, I wanted to just open up an HDF5 file and check it to see if that data reference isn't all zeros, or has tables that I want in there. And the tools that I had weren't good. There was one that was written in Java, and so of course you'd start it up and it would take like three seconds to start. It actually had a bug in it so that it would modify the data to be kind of truncated, it was read-only. And so every kind of tool for quick exploration didn't really work for that purpose. I was really missing that really rapid flow — just show me the data, I want to see it with my own eyes and then pop back out.

Tobias Macey: And then for the sake of you rewriting this tool from scratch, I'm curious if there was any functionality in the original tool that you either consciously left out because it didn't suit your particular needs of the time, or any functionality that you tried to replicate but weren't quite able to match, because you didn't have the necessary context that you had had at the time when you were working at F5?

Saul Pwanson: Yeah, that's a really great question. I've been working on VisiData now for a lot longer than I worked on the original prototype that I had made. But because it was a networking company, one of the things that was in the original version that I still haven't added but would like to add someday, was the ability to add a derivative column, because it was updating live from actual data on the device. For example, if you see the bytes transfer, then you could add another column that would be a derivative of bytes transfer: bytes transfer per second. And you could do that for any column you wanted to, which was a super handy feature.

But I am finding that VisiData is used more for static data now than the original tool was at F5, and so that feature hasn't been added. I do remember that the prototype had pop-ups — for example, if you wanted to change a field between 10 values and enumeration, there was a little modal pop-up that would show up, where you could scroll down and pick the right option. That was always a neat little feature, it made things a little easier to use. Right now in VisiData, if you have to add an aggregator, you look on the bottom line for the list, which I still don't quite like. But, I decided to not add modal dialogues to VisiData. If you want to see something, you have to go to a fresh sheet. If you're going to go to a fresh thing, the modality is just in the bottom status line. That was a conscious design decision. It looks flashier to have the pop-ups, but as I'm doing more design work, I see that modal dialogues kind of get in the way.

Tobias Macey: You mentioned that the common use case for VisiData now is for static data. I'm wondering if it does have any capacity for being able to process continuously updating information, such as the Top command in Linux, or the network streams that you had built the original tool for?

Saul Pwanson: Yeah, absolutely. And in fact, I would love for VisiData to have more adapters or plugins or loaders for other things, like Top. Actually, Top's a great example. And I've got some prototypes of vgit, but vgit's still more static data. I do have a vping, which goes out as a combination trace route and ping, it updates live as it's finding the various hops and their latencies, and stuff like that. But then it turns out that every one of these applications is a pile of work. I've been devoting so much to the core application of VisiData that I found myself not having a lot of time to polish the other ones to the degree that I want to. So I wish that I had that time, or that I could find somebody that wanted to work on it with me. But I've been focusing on the main end of it.

What are the main uses cases for VisiData? [07:22]

Tobias Macey: I'm curious what the main use cases are for VisiData that you have found, both personally and within the community. In particular, which tools has replaced in your toolbox for things like systems administration or data analysis, that you might reach for otherwise, but that VisiData is just more convenient for?

Saul Pwanson: Mm hmm. So it seems like one of the main use cases is to get a first look at downloaded data. I know that Jeremy Singer-Vine, who is the data editor at BuzzFeed, uses it all the time for exactly this purpose. Because you find data all the time online, and you don't know if it's useful, and you don't want to spend a lot of time investing in it — porting it into a database, for instance. We just want to see it right away, to see the first columns, the first rows, do a quick search or however else you want to view it. And to be able to get to that place instantly, as opposed to doing any amount of coding or work, is one of the huge benefits of VisiData. And so I would say that it's mainly super useful as an exploratory tool.

The other thing that I find it super useful for is getting data from one format and structure, into another format and structure. So you know, if you just need this one-off thing — you've got this pile of data here, whether it's an Excel spreadsheet, or even just piped in from another command, and you want to say, remove those four columns, add another computed column, and just save that off and pass it off, you can get all of that done in like 30 seconds. Whereas doing it in any other tool is going to take you at least several minutes to get those tools put together, then write the code, do whatever you're going to do, and it ends up taking more like a half an hour. And so it's a lot of those one-off scripts that I think VisiData has replaced for me.

Tobias Macey: Yeah, there are definitely utilities, both in the general user space libraries — particularly for Linux but also in Python — that can do direct conversion between formats. But the workflow you were describing, of being able to manipulate the information before you commit it to the other format, is definitely something that would typically require a lot more effort and exploration to make sure that you're getting things right. And then, once you do get it working, you're likely to use it repeatedly, but it's going to be much more static and brittle than if you were to use VisiData in a more exploratory fashion. And I know that VisiData also supports being able to build pipelines for that repeatable use case as well. So I'm wondering if you can just talk through an overall typical workflow for data exploration and analysis, and also the sort of conversion workflow that you might use VisiData with?

Saul Pwanson: Yeah, okay. So here's a typical workflow, for instance. I might download some data from the internet, for example from data.boston.gov — every city's got their own public datasets. And so you open it with vd to poke around and see what's there. I like to see the scope of the data and the precision: basically, how wide does it get and how much individual pieces do you get, as well as how clean it is and how useful it is for our purposes. And then, as I'm browsing around, I start finding things — like, oh, this column seems interesting, I wonder what's in here. And because there might be a million rows in the data set, it's hard to see that, especially since it's often sorted in a certain way. And so I do a frequency analysis very often, using Shift+F, which is one of my favorite things about VisiData. So you just use Shift+F on any column, and within a couple of seconds if not instantly, you can see the top value in that column and the number of values in that column total. And I think it's really great for finding any anomalies or outliers in the data. So you can use VisiData as a quick sanity check: "oh, I see, there's really no data in that field, or seldom is there data in that field," or "oh, that's interesting, why are there half the fields completely empty," et cetera. So that's one workflow.

What happens is, once I get the data into a state that I want, then it's pretty easy to revisit the command log in VisiData, cull it down to get only the stuff that I really want, and then save it off. And I can do that repeatedly, if that data might be updated, or I want to share it with somebody else. In fact, one of the things that's been interesting is how useful the command log has been for debugging. People can say, "this is what I've been doing with this, and this is the input data that I have, and here's the command log that I've been running over it." And it's interesting how I can look at the command log, and see, "oh, I wouldn't have known that you did that exact command here, or on that row." And that changes everything. That is the way that I can figure out what's going wrong. So the replay is a really useful debugging aid too.

The command log [11:48]

Tobias Macey: I'm wondering if you can talk a bit more about the command log. As you said, it's definitely frequent that you run through an exploratory cycle, and you finally get to a good state, but you don't necessarily remember all the steps that you've run through. You may have deleted code or added new code, so you don't know exactly what the flow was. Whereas by using a more keyboard oriented tool, you can keep that history and see what the overall workflow was. So I'm curious if you can just discuss the command log and how it manifests in VisiData, and some of the benefits and drawbacks that it might provide.

Saul Pwanson: So, as far as I'm concerned, VisiData is a grand experiment. Originally I made it as just a browsing tool — a CSV browser, as I used to call it. And then it turns out that it's a lot more broadly useful; it turns out that spreadsheets and tables are a very universal construct. I was at the Recurse Center in early 2017, just playing around, and I started to wonder if I can record all actions into a table itself. And I did that, and it didn't work at first, and I was recording all the motions and everything else, and it was a mess. But then once I took everything that didn't belong on the command log out, it worked remarkably well, and I think it's actually super handy. It's not as good, obviously, as a Python script. You can only have one input field, it's kind of a rigid structure. But given the limited amount of data that's on the command log, it's remarkably flexible and powerful. I'm very surprised about that.

There is one thing that I do wish that I could solve, and I'm sure that it is solvable, but I haven't managed it yet. The command log reports every action that you take, including all the dead ends that you wind up finding yourself in. Those are sometimes handy to have on there, so you can see what you did and didn't want to keep around. But when you're getting to the end, and you want to do it again, you want to cull all those dead ends out, and just get to the place that you currently are. And so I want to have some kind of graph or tree that would get you from your current state and only show the commands that you took to get there. That's what I really want to add to VisiData, and I just haven't gotten to that point yet.

Tobias Macey: And another thing that you mentioned, which I also noted in the documentation, is the case where you have a file, which might have millions of lines, which would typically be either difficult or impossible to open in a more traditional terminal oriented editor, such as BM or Emacs or the less pager. And I'm curious what types of performance strategies and techniques you've used to be able to handle files like that, and particularly, being able to manipulate them without just exploding your memory usage.

Saul Pwanson: So one thing is that VisiData actually does explode memory usage. It doesn't explode as much as some other things, but in my mind, it actually does take up a lot of memory. But beyond that, though, I think there's a couple things that matter here. One is that there's an asyncthread decorator that you can add to any function in VisiData. And that means that when you call that function, it spawns a thread and goes off and does it asynchronously. And there's always one main interface thread that keeps active, that is constantly updating with its status. And then within that thread, you can add a little progress meter, and those are pretty easy to do. But the main thing is that because it is so easy to spawn additional threads, I do it all the time, whenever I'm doing anything that might take a while. And I know that any kind of linear operation on data, because you might have millions of rows, might take a while, so I want to make each operation spin off its own thread. And that keeps me conscious of how much time things are taking, for one thing.

But I want to say that I actually don't think VisiData is fast, in itself. It's just responsive, and it turns out responsiveness matters more than speed. I would rather spend 10 seconds seeing a progress meter update and make it to the end, than five minutes with no progress and no knowledge of how long it's going to take at all. The first one is kind of soothing — I can take a break for 10 seconds — and the second one is very frustrating and makes me almost on edge. Like, do I need to do something here in order to kill it before it takes over my entire computer.

The other thing that's important is that tools like Excel or vim often want to own the data. They want to import it into their own structure and format, and that's what causes the thing to really blow up. Every cell has to be stored separately in its own custom thing. Whereas the key architecture thing that VisiData has that makes it so flexible is that it stores the rows natively. And so whatever I get from whatever Python library, that just becomes the row. Every item is just that object. And then columns are basically lenses into those rows. And so VisiData is computing the cells every time. It doesn't grab the cells and put them in, it just goes ahead and computes it whenever you want to see it. And it's adding a column, trivially – you can add a column in constant time. And that also means that saving, for instance, is comparatively slow in VisiData, because it means it has to do the evaluation of all those columns for every single cell when it's saving. But it turns out, you don't actually have to save everything, usually — you're not actually looking at everything, only looking at the first hundred rows, or a few columns, or whatever. So I feel like between the threading and the conception of rows as objects and columns as architecture, that's what really helps VisiData stay focused on the user experience like that.

How has VisiData evolved over time? [17:12]

Tobias Macey: Can you dig a bit more into the architecture and implementation of VisiData itself and some of the ways that it has evolved since you first started working on it? And I know that you've also got an upcoming 2.0 release, so maybe talk a bit about some of the evolution that's coming there, and then any libraries or language features that you have leaned particularly heavy on and found most useful in your work.

Saul Pwanson: Probably the biggest evolution is that, when I first was doing it, it actually started out in a single file script, I put everything into a single vdtui.py. And the idea was that it was a drop-in thing: you could copy it over to a server over SSH, and then you could start using it, and you wouldn't need anything but the base Python. And I licensed that as MIT. And then as I started adding more modules and internal plugins to it, I licensed those as GPL-3. I was trying to keep this very clean separation, because the idea was that you could use the vdtui, which was very similar to the thing I had at F5, for making all kinds of other apps. But then nobody really took me up on that, and it's kind of hard to use somebody's single file library like that unless it's a super tight little library. So I kind of gave up on that, and am now heading more towards a more plugin platform architecture, where VisiData the app is the thing that hosts the individual plugins that you can add. It may even wind up having a vgit application that you can use. But it's kind of turning on its head. As opposed to incorporating vdtui in some particular version of your thing, you're actually using the VisiData library.

Beyond that, it's now just a bigger open source project. It's got a whole packaging release cycle. And I'm working with Anja, who has been very instrumental in some of the packaging, testing, and documentation stuff that we've been doing. And it's just taken off and gained more traction in that sense. You also asked about the libraries that I like. One of the things that I'm doing to keep performance good is that I take very few dependencies. I feel like layers are how things get messy. So the fewer layers that you have, the better off it'll be, if you can wind up coding everything in between there. So as far as the libraries that I use, obviously curses is essential, but that's built into the Python standard library. The Python standard library is really fantastic, and everything's included, which is a super bonus. But then also, the PyPi ecosystem in general is so broad, that any format that I come across, HDF5 or Excel or whatever, they have a library already for it. And it's a library that you can use in Python, you write a page of code and you've got the stuff in there. And then all the loaders are just importing those libraries, then calling them, and putting the rows and the return in there and having some columns around — it's a nice, pretty simple concept.

One thing you mentioned was that you wanted to know other ways that I keep VisiData fast. I've been very focused on making sure that it starts up very quickly. I feel like, if there's a half second of startup time, it just gets in the way and it feels like a certain kind of friction. And so one thing that I do a lot of is lazy importing. For all these libraries, I have no idea how long they're going to take to load or start up themselves. And I know that there are some pretty heavy ones that VisiData uses sometimes, but I don't use those unless I need to. When you open up an Excel file, for instance, that's the point where it imports the XLS library. And if you don't ever load an Excel file, then it doesn't have to spend the time doing that. So that's another one of those tricks.

Before we move on, I wanted to mention the Python dateutil library. I don't take many dependencies, but that's one that I've been very happy to take, because it parses any date format that you can throw at it. It's amazing. It's a best in class detection and parsing tool. Also, another feature that I have used a lot is Python decorators. I think that's a pretty standard thing, actually, but I use them a way of tagging functions. For instance, I mentioned the asyncthread decorator, which takes a pretty advanced concept and reduces it to just the essence, so that I don't have to think or work hard to have those concepts work for me.

Tobias Macey: Yeah, the decorator's capabilities and the syntax that allows for it definitely makes it a lot easier to organize code and concepts. You can just drop it on top of a function definition without having to try and incorporate it into the body of the function and remember, "what are the return types? What are the inputs?" You just say, "it wraps this function and then it handles it, I don't have to think about it anymore." And you can just keep it all in a utility's library, for instance.

Saul Pwanson: Yeah, exactly. Totally. I will say that one of the things that I wish the decorators did support was to be able to put the def function, name and signature on the same line as the decorator. It's a minor point, but I use a lot of grep type tools, and I would like to be able to know that function that I'm looking at is an asyncthread, or has a deprecated decorator now, and be able to see that when I'm searching for functions, for instance.

Tobias Macey: Well, one tip there is that if you add the grep -b1 that it'll show you the line that you want, as well as the line just before it.

Saul Pwanson: That's a great tip, thank you.

Why the terminal? [22:36]

Tobias Macey: Another thing that stands out with VisiData, particularly, is the fact that it is entirely terminal-oriented, whereas a lot of data exploration tools will be more focused on a graphical interface, or trying to embed into a Jupyter notebook for providing some sort of visualization. And I'm wondering, what was your motivation for focusing so closely on that terminal interface and making it a command line client?

Saul Pwanson: I feel like the terminal is my home. I've been using terminal since the '80s, and so I'm very comfortable in that environment with those kinds of restrictions. I'm also much more comfortable with the keyboard than a mouse, but as I am getting older, it's becoming harder to just type verbosely. So I wanted individual keystrokes to do things, and I couldn't figure out how to do that in, for instance, a Jupyter notebook. The other thing is that because the terminal is so old, and mature, let's say, it's a universal interface. Any platform has an SSH client and a terminal. The only other way that you can get that level of universality is with a web browser and, for instance, an electron client, and that's way too heavy. So I feel like the choice is between a terminal which is very light, and an electron client which is very heavy. That choice for me is obvious. I want it to be a very quick in and out thing. I don't know how I would do the same kinds of things as quickly in the web, or even in a native app, if you have to reach for the mouse. You can do it with a with a given native app for sure. But then you have to make a native app for Mac, for Windows, for Linux, et cetera, and I just didn't want to do that. And so really, VisiData is about only about 10,000 lines of code all told — which is actually quite a bit, but it's not that much when you compare it to comparable tools like Open Refine. And I think that's a testament to being on the command line. I can do the minimum necessary to get the job done, and don't have to worry about things like pixel width. It's like, "No, you choose your own font, you choose your own font size and ways to interact with the thing — I'm just here to get out of the way."

Tobias Macey: It also has the benefit that, as you were describing originally, you can just copy it over via SSH, or now pip install it on a remote machine. You don't have to go through the hoops of trying to set up a way to have a graphical interface to that remote box, you can just copy it over. And it broadens the reach and capabilities and use cases for the tool, where the only access you have is via terminal, which as you said, despite its age is still a fairly common experience for people who are working in technology.

Saul Pwanson: Yeah, absolutely. I work remotely now, and we do have screen-sharing apps, but they don't always work that great, and you have to sometimes install some other plugin. The app that I love most, is called tmate — it's a tmux wrapper, I guess. And you just install it, and you type tmate, and you get an instant shell into your own machine and you can give somebody an SSH link. And they can do that, and people are usually amazed. Between that and VisiData, now we have an instant data exploration platform we can share, alongside a chat client or whatever. That's it. And I find that to be so much more accessible than modern video chat platforms — even us, at the start of our session here, had technology difficulties. The shell is very reliable, by comparison.

Tobias Macey: By the fact that you are targeting the terminal environment, what are some of the constraints that that brings with it, in terms of your capabilities that you can bring into VisiData? What are some of the most challenging aspects of trying to build a user interface for data exploration and analysis, within this environment that is so graphically constrained, particularly given the fact that you have incorporated some visualization capabilities, and just some of the ways that that manifests?

Saul Pwanson: I have to say that I've been pleasantly surprised that I haven't felt as constrained as I thought I would. For example, once I discovered that I could use Braille characters to do the graphing, it kind of just worked. It's not perfect, but if you want more perfect things, you should be using other tools. And in fact, that's one of the things I think that is important with VisiData, is that it's not meant to be a be-all and end-all. It's kind of a glue technology, right? It's an adapter. And so, once you figure out what you want to do, then you should go to the fancier tool and do it right. But there's no reason to do everything super right from the get go. You just want a quick glance at it.

You asked about the most challenging aspects of building a terminal UI. And I have to say, the thing that's been most challenging for me is knowing that if VisiData were on the web, it would probably be worth a lot of money, and doing it in the terminal means that I've kind of eschewed that. No one really pays for terminal tools. I shouldn't say no one — I actually have several Patreon subscribers, and I'm really appreciative of them.

But you know, if you think about how VisiCalc back in the day was a fancy program that sold for hundreds of dollars, I can't imagine doing that with VisiData. That's just not how the world works. Although, if it was a native app, that might make it possible to sell VisiData, for instance. The other thing that is more challenging than you might think, is that you mentioned pip installing something. And that's really great for people who already have Python and already know how to use pip. But I actually think VisiData is a pretty reasonable tool to use for anybody — anybody who's willing to use the keyboard anyway. And yet, installation is one of the trickier parts, right? If I wanted my partner to go and install it on her computer, I have to tell her "install this, click here, download this." People just want a single thing they can download, they can double click on, and then go. And that's just not how the terminal world generally works. So I feel like installation is one of those weird things, where there's a larger barrier to entry than there should be, and yet I can't find a super easy way around that. So that's just how it has to be.

Tobias Macey: I'm also wondering which terminal environments it supports. Because Windows is generally one that's left out of the support matrix for a command line tool, but because of the fact that Python does have the curses interface built in, or if you are relying on the prompt toolkit library, I know that there is the possibility of being able to support the Windows command lines, I'm just curious what the overall support matrix is for VisiData.

Saul Pwanson: I started off doing it in Linux, because that's what I run, and then it turns out that it works on a Mac terminal just fine. And I didn't have access to a Windows machine, and so I was like, well, Windows isn't fully supported. It turned out that people were just running it under Windows subsystem — Windows Subsystem for Linux, WSL, I think is what it's called? And it basically just worked there. Then somebody submitted a small patch, and it works even without WSL now, I think, on a more recent version of Windows. So I actually have never used it on Windows, but I know we've got quite a few people. Some Italian open beta users, like Andrea Borruso, love using it on Windows, and it works fine. So as far as I'm concerned, it works on Mac, Windows, and Linux and seems to be fine on all of those. So I wouldn't say those platforms are what we support, necessarily. But if it works, I'm not gonna say we don't support it either, right?

The design of keybindings [29:54]

Tobias Macey: [chuckles] Another peculiarity of building a command line oriented client is that it is heavily keyboard-driven, as you mentioned, and that means that you need to create the set of key bindings that will do whatever it is that you wanted to do. So I'm wondering how you've approached the overall design and selection of those key bindings, to ensure that there is a set of internal consistencies, and that the key bindings make semantic sense, but also so that you don't run out of key bindings in the event that you have some new capability that you want to run into, because it is a limited space, even when you do incorporate modifier keys.

Saul Pwanson: Yeah, totally. We're running up against that now. There are few keys left, in some sense — at least, few keys that people want to use. So the main thing is that I have to use mnemonics to make sure that key bindings stick in my own memory. I actually have a pretty bad memory myself, and so if they don't fit my mental model, I can't remember them from one month to the next. So I've made sure that they at least make sense to me. And because I've been using terminal stuff for so long, I'm already tuned into the the existing text culture that is around. So a lot of the key bindings are borrowed from vim, like d for delete, a for add, and so on.

Actually, when I showed it to a guy on Ingy döt Net, he chastised me for the fact that Ctrl+Z didn't suspend in VisiData, and I was like, "Oh, you are totally right, that needs to be fixed right away." And I did fix it right away, because you want to make sure that the things that people are used to will still work. And so Ctrl+C will just work. The other thing I think is really important is that there's layers of mnemonics. We have a couple of modifier keys, and I wanted to keep those very simple. Like, I think that vim is great, but it feels daunting when you see all the different possible combinations.

And so I hope that when you see that VisiData has exactly two prefix keys, that it feels like "okay, maybe we can wrap our heads around this." And those are just modifiers on other existing commands. And so there's layers. Another piece of this is that column transreferences, for instance, are all on the symbol keys. And in fact, all the types are all on a single row. And so you've got date is on @, and converting into an int is on the # sign. Those are all just adjacent to each other on the top row of the keyboard, and then other column things are also symbols.

And so you don't get confused thinking well, "Isn't an int i?" "– No, it's one of those ones up there." "Which one is it? Oh, I think it might be the one that looks like that." That helps me, anyway. And then everything that's for going to a new sheet is all on the Shift key. For me, it's like shift and sheet almost rhyme. So Shift+F goes to the frequency sheet, stuff like that.

And then finally there's symmetry, I think that's really important. So I reserved all of the pairs. For instance, open paren (, close paren ), open brace {, close brace }, those were all reserved from the get go, for things that have both a front and a back. So sorting is on the square brackets, and one way goes ascending and the other way goes descending. Scrolling down to the next item is a greater-than sign, versus the previous item, less-than sign. And I feel like symmetry on those things is very useful, but then also, more largely, symmetry between commands. So the g prefix goes bigger, and the z prefix is smaller, more precise. And so when you say "I want to delete all the rows I've got selected." That's gd. "I want to unselect all rows," that's gu. To me, in a way, it feels like what makes sense. You may not know it before you discover it, but once you discover it, then it's sticky. I feel like that's a super important thing. So, the VisiData interface isn't made for the first time user, it's made for, like, the third time user.

Tobias Macey: Yeah, there are definitely a lot of peculiarities, and a sort of culture and history built up around different key bindings. And as you mentioned, vim has its own set, Emacs has a different set of key bindings that people will be familiar with, and then there are any number of command line tools, that have all created this sort of general pattern of how you would craft these key bindings. So it's definitely interesting to hear some of the history of how you have approached it, because of your particular toolset choices. And like you said, anybody who's been living on the terminal long enough will find it fairly natural. I appreciate the care that you've put into considering how you add new key bindings so that it doesn't just end up cluttered and so that it – so that you can have some sort of mnemonic muscle memory of being able to recreate a certain workflow once you pick it up. Because with any tool, there will be periods where you put it down for a while and don't come back to it. And then when you do come back to it, you want to be able to just get right back to being productive without having to go back to try and remember what were all the key commands, and look at the reference manual.

Saul Pwanson: Yeah, totally. The other thing that we did was that a year ago or so we instituted longnames. So originally the key bindings was all we had, for example, Shift+F means the frequency table, and that was the only way you could access it. But now there's actually a longname for that, I think it's open-frequency-table. I'm not sure that's exactly right, don't quote me on that. But then you bind Shift+F to that, so people can rearrange their keyboard, if they really want to, but it also gives them the ability to add commands that aren't bound to any key. For instance, if you make your own command in your ~/.visidatarc, you can make that and bind it yourself. Or if we create a command that's very unused — for instance, random rows. That used to be on Shift+R, but then we made Shift+R be redo, in the undo-redo pair, because that's kind of a symmetry between Shift+U and Shift+R, as opposed to vim's lowercase u, Control+R, which doesn't make natural sense, to me anyway. Anyway, I moved random rows off of Shift+R, but then there's no real good place for the random selection of rows to go. So since I feel like selecting random rows is an infrequent operation, I put it on a long name, so then you can just press the spacebar and enter in a long name, and it goes ahead and does it. So that's another tactic, is to start moving things off of the default key bindings, into a huge list of possible commands that you could use.

Tobias Macey: Yeah, that's another thing that's got a long tradition, both in vim and Emacs and other tools — being able to have a way of opening a prompt, so that you can then start typing, as you said, a long-named command, or being able to start typing it and then maybe tab through to cycle through what the commands are, so that you don't have to necessarily remember all those off the top of your head as well.

Saul Pwanson: Mm-hmm, totally. We do have tab completion, and it works anywhere that we have open prompt, including the long name thing.

VisiData's capabilities [36:37]

Tobias Macey: Then, in terms of the types of analysis that VisiData can do out of the box, I know that you mentioned frequency analysis or histograms. But I'm curious, what are some of the other capabilities that come natively in VisiData, and any of the interesting plugins that you or others have contributed for being able to expand the capabilities and utility of VisiData?

Saul Pwanson: So, out of the box, it can do all kinds of interesting stuff: searching and filtering, bulk editing and cleaning, spot checking, finding outliers. I use it, actually more often than I would think, for file format conversion. The ability to load any format and then save it to JSON or character separated value or Markdown is super handy, and it gets me from here to there a lot faster than I could otherwise. Even scraping a web page for its tables is basically built-in. Jeremy Singer-Vine, for instance, has written several plugins already for the current version. He wrote one that does row duplication, and a loader for the FEC, the Federal Election Commission dataset, and you just download those and import them in your VisiData RC, and they're ready for action right away.

Tobias Macey: And when I was looking at the documentation, it seems that one of the libraries that you can load into it is pandas, and I'm wondering if that means that you can expose all of the pandas capabilities as well, as you're exploring these data sets. Because I know that that's often the tool people will reach for, their first time digging into a data set just to see what's the shape of it. And so I'm curious how that works into the overall sort of use case of VisiData as the exploratory tool, and then where the boundaries are, when you might want to jump to pandas, or if you could just incorporate that whole flow together.

Saul Pwanson: Yeah, that's interesting. I made a very simple adapter for pandas, it literally was maybe 20 lines of code at first, just because pandas supports a lot of different loaders, too, and it's super handy to be able to use those and browse those. But what's interesting is that pandas and VisiData don't actually play that well together. In order to do things like sorting, for instance, VisiData grabs each value and sorts based on that, but pandas' built-in sort function does it more efficiently. And there's just no good way to do that automatically. You have to write all the commands in a way that's compatible with pandas for pandas sheets. And that's totally doable, but it's a fair amount of work, and I haven't done it. Somebody did make some modifications to make pandas more responsive in certain cases, to make things work better. And that's totally doable. Like I said, it takes a fair amount of work, and it doesn't happen naturally. You can't just use pandas' things like you'd think. You can use some of the functions that pandas has, on pandas sheets, and even on non-pandas sheets if they're standalone functions, but to use a pandas dataframe just naturally like you would, you probably are better off using it in Jupyter by itself.

The community [39:40]

Tobias Macey: In terms of the overall growth and adoption of VisiData, it seems that there's a decent community that's grown up around it. And I'm wondering how you approach the project governance and sustainability as a solo developer, and how you are looking to grow the community and incorporate more people into the future of VisiData.

Saul Pwanson: Yeah. Well, you're saying I'm a solo developer, but I've got a little bit of help now. Like I said, Anja has been instrumental in helping me with decisions and discussions, stuff like that. There's also a #visidata channel on freenode that several of us are hanging out in, where people can talk about things and ask questions. I prefer personally a chat system like that, because I find myself doing a lot better with chat, which I've been on chat platforms for over 20 years now, than I do with email. Email is a lot heavier, it requires more intention and attention, and with chat I can just toss up an answer and it's just done.

So I'm, of course, the decider on those things. But I have to be honest, it kind of feels like I'm discovering VisiData more than creating it at this point. It's like a chunk of marble to a sculptor, it tells me what it wants to become. And there are some things that I didn't even consider, and then I look at it like, why didn't I think of that already? For example, the rowtype down in the lower left corner, where it shows you lines or columns or whatever the current data type is. For the longest time, almost to 1.0, that just said, "rows". And I didn't know why I even put the text there, if all it was going to do is say the same thing every time. And yet I felt strongly that it should be there. And then once I realized that should just be the rowtype, I was like, oh! And I don't feel like that was my creation. That's just how it had to be, if that makes sense.

So there's that, and then you mentioned about project sustainability. The thing is, my energy is my most precious resource — my energy and ability to code. I have a day job, and so I come home at night, and it's then that I want to screw around with VisiData. But it's really hard to summon the energy when I don't have a concrete use case, or someone that really cares about something. So I have the most energy when somebody is around and is enthusiastic, and they have a sample data set, and they're like, I just want to do this thing to it. It becomes kind of a little puzzle you can put together. Can I use this existing command to do this, or here's a one-liner to put in their ~/.visidatarc, or does this require a different core piece of functionality — so that now, not just that case, but ten other cases can be solved too. Those are the things I enjoy the most, I actually do really enjoy solving those puzzles. But then sometimes we'll have people who ask for a generic feature, and it doesn't feel like it's very immediate, it's more abstract. Or if I have a concept for something that I've been wanting for a while, and because there's nobody who really, really wants it, I get less motivated, and I just kind of decide to do something else.

Innovative uses of VisiData [42:51]

Tobias Macey: So what are some of the most interesting or unexpected or innovative ways that you've seen VisiData used?

Saul Pwanson: I feel like we have a couple of superfans, people who will use it for well more than they really should. One of them is a guy named Christian Warden, and he does a lot of Salesforce consulting and stuff. So he's got buckets of data, and just wants to move through it quickly. He built a duplicate row finder for some dataset with Python expressions and a .vd script. VisiData is not made for internal computation; you can have elements that compute within a row, no problem, but if you want to look at the previous row, it's not really meant for that. I mean, I'd like to add that at some point, but I haven't figured out a really great way to do it yet. But he figured out how to pull it off, and it was an amazing beast, and it worked. It actually exposed a bug with computation, so that it was taking forever to run. But once he fixed that, it actually was remarkable, like, wow, you really have turned this into yet another Turing-complete programming environment. [laughs] So that has been kinda weird.

Also, I'm not sure if you or your listeners have seen the lightning talk that I gave a couple years ago, but I had some data that had lat-long coordinates, and I was just curious if I could plot those in my little canvas. And it turns out that plotting latitude-longitude as x,y coordinates, works really well for maps, even for like a million points, there you go, you can see the distribution of things. It was surprising to me that it worked as well as it did to be honest. Like, I don't think this is built for geographic information at all, and yet you can kind of pull it off. So that's been both surprising and unexpected and – yeah, kinda pleasing, too.

Tobias Macey: Yeah, I did see that lightning talk. And that was one of the things that I was kind of blown away by as far as the visualization aspect of VisiData, given that it's a terminal environment. And so it's interesting to hear how you just mapped the lat-long to x,y coordinates. And I'm sure that you just figured out what were the maximum bounds of the coordinates that you had to figure out, what the overall plane coordinates needed to be in relation to each other, so, that's pretty funny.

Saul Pwanson: Yeah, thank you. One more thing, if you don't mind, that has been kind of surprising to me is how meta the thing goes. So editing VisiData's internals, using VisiData's own commands, is something that's been kind of surprising for me. Just yesterday, a user asked how they could get a type column on the Describe Sheet. And I thought about it and I was like, you know what you can do, is: you can go to the Columns sheet, and you can copy it from there, and then you can paste it onto the Columns sheet of the Describe Sheet. And it'll just work. And it's like, you couldn't possibly do that with Excel. Similarly, if you've got 1000 columns, and you want to search or select all the ones that begin with a certain thing, and remove all those from the set, you can do that in VisiData. And that's no problem, it just works just like anything else. I have no idea how you do that in almost any other tool. And so I feel like the metadata editing aspect of it has been surprising for me, even though I put it in there, but the fact that it works as well as it does has been really kind of interesting.

The future [46:08]

Tobias Macey: And looking forward, what are some of the features or improvements that you have planned for the future of VisiData?

Saul Pwanson: So right now we're working on the 2.0 release, which will be still a couple months out, and the goal is to stabilize it. The current version, 1.5.2, is actually incredibly stable, and really tight — there's at least one bug I've seen, but that's fine, it was an edge case. But the API is all over the place, it's not as coherent as the user interface. And so one of the things we will want to do is make the API stable, and produce some more coherent documentation about the internals, we're calling it the "Book of VisiData." And the point of that is so that we can let it rest and work on some other things, but we can let other people go wild and share their own creations — plugins, commands, loaders, or whatever, without destabilizing the core goodness that is there. So I'm sure there will be a 2.1 or whatever after that. But I'm really hoping that after 2.0, development can slow down, and I can move on to some other projects that I have in the queue.

So one of the things that we've been talking about that's gotten a bit of traction is something I've been calling "Where in the Data is Carmen San Mateo?", which is a throwback to an old game from the 80s and 90s, which maybe you've heard of, called "Where in the World is Carmen San Diego?"

Tobias Macey: I used to play it, yep.

Saul Pwanson: Okay, yeah, so you're familiar. So the idea is that I want that kind of game, but with data and data sets. So it'll be for hardcore data nerds, kind of an escape-the-room game, or a choose-your-own-adventure kind of game, where you're solving a crime, but you get data sets to look at, and that's how you get the actual clues and solve the puzzles. I'd like to work on that, and so that's my next project, probably, in the queue. But I don't really want to do that until we've got VisiData 2.0 locked down, and we feel like it's a stable place for everybody.

The terminal renaissance [48:09]

Tobias Macey: Are there any other aspects of VisiData or data exploration or any of the other accompanying projects that we didn't discuss yet, that you'd like to cover before we close out the show?

Saul Pwanson: Not specifically, although I did want to say that I feel like we're in an age of kind of a terminal renaissance. We went through the period of the late '90s and early 2000s, where it was more graphics, all the time. And that was the obvious way up and out. But the terminal has been with us throughout, and I definitely have never left it. I feel like within the past, maybe 10 years or so, with projects like ohmyzsh and tmux/tmate and many other ones, that the terminal has been kind of getting a resurgence. And even now, when people go to data science boot camps and stuff, they have to learn the terminal and get involved in there too, because you need to be able to do that in order to dive deeply into data stuff. And so I feel like this is part of that. VisiData is saying no, wait a second, you don't have to be in the web graphics world in order to be high perfection — in fact, not being in that world makes it a lot easier for you, if you can just embrace the fact that you're going to be at the terminal and using a keyboard.

Tobias Macey: Yeah, I definitely appreciate the fact that there is a lot more focus being paid to just making things that work in the command line and being able to stitch them together. Because, as you said, the graphical interfaces – well, they are appealing, and it's easier to sell something to somebody who isn't as technical, if you're in that environment – they do bring a lot of extra weight and requirement to the development and maintenance of them, as well as, in some cases, the use of them, because they are definitely much more mouse -riven. And it makes it harder to be able to have just a unified flow.

Saul Pwanson: Yeah. And you know, to be honest, the versions of iOS and Windows are going to keep marching forward. And I have no doubt that if I made an app for the current version of either of those things, that in the next three or four years, it wouldn't work with the next version. I am actually pretty confident that if I don't touch VisiData for the next four years, that you'll be able to use it in the next version of Python, on whatever platform, no problem at all. And I find that to be really motivating to do a good job now, because I don't have to keep writing it, I have to do it good once.

Closing [50:34]

Tobias Macey: For anybody who wants to follow along with the work that you're doing or get in touch, I'll have you add your preferred contact information to the show notes. And with that, I'm going to move us into the picks, and this week I'm going to choose a newsletter that I actually found while I was doing some research for this conversation, called Data is Plural. It's maintained by Jeremy Singer-Vine, who you mentioned a few times. It's a weekly newsletter with different interesting data sets that looks to have some fairly curious discoveries. So if you're looking for something to experiment with VisiData, you might have some interesting finds in there. So with that, I'll pass it to you, Saul, do you have any picks this week?

Saul Pwanson: I wanted to promote tmate, which I think I have during this episode already. Definitely give tmate a look, if you're a terminal user and want to have a multi-user experience; there are a lot of other tools in that same vein like mosh, I want to give a shoutout to, the mobile shell for less than perfect network connections. And – yeah, there's all kinds of good tools out there, but I'm not sure I can come up with any more off the top of my head.

Tobias Macey: All right, well, thank you very much for taking the time today to join me and share the work that you've been doing with VisiData. It's definitely something that I'm going to be experimenting with because I spend a fair amount of my time on the terminal, and have to do a lot of exploration of random data sets, whether it's just .csv's or piping things from different batch commands. So thank you for all of your work on that tool, and I hope you enjoy the rest of your day!

Saul Pwanson: Awesome, thank you very much! I hope you have a great day too.

Outro [52:06]

[Outro music begins.] Tobias Macey: Thank you for listening. Don't forget to check out our other show, the Data Engineering Podcast, at dataengineeringpodcast.com, for the latest on modern data management. And visit the site at pythonpodcast.com to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something, or tried out a project from the show, then tell us about it. Email host@podcastinit.com with your story. To help other people find the show, please leave a review on iTunes and tell your friends and coworkers. [Outro music ends.]

The very last VisiData TinyLetter!

2018-12-26T00:00:00Z

[originally published as TinyLetter #16]

Hello VisiData enthusiast,

Thanks for signing up for this TinyLetter and for your continued support. After some reflection I've decided to move these kinds of updates to my Patreon, and I would encourage you to check out the most recent one, which talks about v1.5.1, a forking undo experiment, and saving to sqlite:

https://www.patreon.com/posts/v1-5-1-released-23468786

If VisiData is your jam then I would also encourage you to subscribe, because there will be some pretty interesting things coming in the next year. But all posts will be made public, so you don't even have to subscribe in order to stay updated.

Happy new year, and please come join me on Patreon!

Saul

Version 1.3.1 released; is this the last TinyLetter?

2018-08-31T00:00:00Z

[originally published as TinyLetter #15]

Hello VisiDaters!

The past few months since v1.2 have been a whirlwind. We've completely reworked the commands and options, created a new keyboard layouts page, and added some new power features. For full details, please see the release notes for v1.3.

This was a particularly intense release because it involved substantial changes to the underlying architecture. We have already sent out a patch for the most pressing issues and are now heading towards a v1.3.2 release, since we've found and fixed a couple more bugs since then.

The plan is to release v1.3.2 in a couple weeks. Due to the scope of the next batch of planned features, along with me transitioning to a new job starting next month, the next release may not be for quite awhile. So please give v1.3.1 your diligent attention, and if you notice and bugs or nagging issues, let us know by filing an issue and we'll do our best to fix them for 1.3.2 so that our collective VisiData experience is maximally awesome.

Finally, I'm thinking of retiring this TinyLetter. I've been sending these letters for about a year now, and this kind of 'marketing' output is draining for me and does not seem to provide any clear benefit. (Does anyone get any value from these letters? I don't get responses to my requests for feedback so I have no way of knowing.) If anyone wants to help out with crafting these monthly updates (or posting tidbits to twitter, or reddit, or whatever), that would be very welcome! Otherwise we'll just drop it and hopefully our other channels (github releases, website, maybe twitter) will be sufficient to notify people of the goings-on.

VisiData your heart out!

Saul

I'll show you my .visidatarc if you show me yours

2018-07-24T00:00:00Z

[originally published as TinyLetter #14]

Yo VisiDaters,

Do you have a .visidatarc file? It's where you can set options and add your own commands, keybindings, and functions to use in expressions--super handy for fine-tuning your personal experience of VisiData. If you have one, I would love to see it. Please send it to me [vd@saul.pw] and if there's any common trends, it may even inspire changing the defaults.

Additionally, I wanted to demonstrate the cmdlog history, which is toggled by an option. If you add this to your .visidatarc:

 options.cmdlog_histfile = "~/.visidata/history.vd"

it will append each command to the cmdlog also to that file. Then you can press gD to go to the .visidata directory, and ENTER to visit the history. I've used this with zY (syscopy-cell) to copy/paste a complicated regex I had used a session or two before.

Finally, we are considering dropping the menu system. Does anyone use and/or like it? Anja opened up an issue on Github and we'd love your comments: https://github.com/saulpw/visidata/issues/174

Keep an eye out for v1.3 in August!

Happy VisiDating, Saul

VisiData: Video #6 (Big Melt); keyboard cheatsheet; the Great Renaming

2018-06-21T00:00:00Z

[originally published as TinyLetter #13]

Been awhile since I rapped at ya! I started a job last week, so I'm going to keep this short, but I wanted to let you know about a few things:

Video #6 is out, and this time it's all about Big Melt ('gM'), a variant of Melt ('M') that was added to 1.2:

https://youtu.be/9RndW_yC45U

As we're talking about adding an 'undo' command and which key to put it on, Daniel E. (@deinspanjer) had the great idea to make a keyboard cheatsheet. If you're interested in following the discussion, here it is:

https://github.com/saulpw/visidata/issues/160

Finally, the longnames are going to be renamed in v1.3, probably to what will become their final home. These are what will appear in the cmdlog (instead of keystrokes), and used for keybindings, macros, etc. If anyone is interested in reviewing these names for intuitive conceptual integrity, please email me! I'd love to have some more eyes on this endeavor.

Happy Summer VisiDating, Saul vd@saul.pw

PyCon2018 and Video #5

2018-05-19T00:00:00Z

[originally published as TinyLetter #12]

Greetings VisiDaters!

I'm back from PyCon2018, but still recovering, so I'll keep this one short.

First, PyCon happened! There was no lightning talk this time, but we had both an "Open Space" and a sprint for VisiData, and it was great to engage directly with users. Several fixes and documentation improvements are in the works because of suggestions we got during these sessions. Thanks in particular to Rob, John, and Anja for sprinting on a yaml loader and the new rows-[un]select-expr commands, both of which will be available in v1.3.

Second, there's a new Twitter handle in town! @VisiData ~~(note the underscore)~~ is now the official VisiData account to follow. I migrated my personal account (and recreated a new personal account), so you won't have to do anything if you're already following me. But it's a lot clearer now which tweets should go where.

And finally, here's video case study #5, in which I reshape an html table using a bunch of nifty commands, including fill ('f'), melt ('M'), and pivot ('W'): https://www.youtube.com/watch?v=0iJ9F4uTkz8

Okay, I'm off to sleep off some more jetlag. Thanks for using VisiData, and as always, feel free to send me an email, I'd love to hear how you're using it!

Saul vd@saul.pw

Version 1.2 released; video #4 describes a major new feature

2018-05-03T00:00:00Z

[originally published as TinyLetter #11]

Hello VisiDaters!

There are a couple of items this time.

First, we have released v1.2! It is currently up on pypi, and is making its way to all of our other distributions. Here are the highlights (the release notes for v1.2 can be found at https://github.com/saulpw/visidata/releases/tag/v1.2):

an additional install method: conda, via the conda-forge channel (https://github.com/saulpw/visidata#install-via-conda)
expanded commands such as:
- g^S (multisave all sheets),
- g* (replace selected row cells with regex transform)
- gM (melt using regex capture groups to transform columns into Variable-Value-Value... rows).
a prototype for a new macro system. On the CommandLog, z^S will allow you save selected rows to a macro for a given keystroke. These macros will be saved to the .visidata folder and loaded automatically on startup. Give it a try, and please let me know what does or does not work for you!
new loaders for sas7bda, xpt, sav and dta (SAS, Stata, and SPSS formats)
and last but certainly not least, an expansion of the directory browser. Users can now modify file metadata, move files around, and delete files, all within the existing VisiData framework.

This last feature is so powerful that we made a video demonstrating it. Video #4 (https://www.youtube.com/watch?v=l2Bpmm0yAGw) is up on Youtube.

Secondly, I've been so heartwarmed by those of you who have reached out recently to share what you love about VisiData. It's really inspiring!

In particular, a couple of you were so happy that VisiData does not save anything to disk or use any swap or tmp files, without explictly executing one of a very small set of commands--only ^S ^D ^O (save data, record cmdlog, open external). This had not been an explicit design decision, but I can definitely see its value, so we will do our best to preserve it.

This feedback was timely, however, as there is now "command history autosave" in v1.2. One source of frustration I've had with VisiData, is that I sometimes find a great workflow and then quit out entirely mere seconds before realizing I wanted to save the commandlog. But taking the above design decision into account, it is disabled by default. If you want to give it a try yourself, you can put this in your .visidatarc:

options.cmdlog_histfile='~/.visidata/history.vd'

Next week, we will be in Cleveland at PyCon. If you will be there, send me an email and let me know! I would love to say hi and chat for a bit.

Au revoir (for at least a couple weeks), Saul

VisiData Case Study #3: graphing calculator

2018-04-16T00:00:00Z

[originally published as TinyLetter #10]

Hello fellow VisiDaters!

Video case study #3 is now available. This newest installment focuses on creating your own datasets out of nothing: adding blank sheets/rows/columns, mass editing, setting a column to a Python range, and all under the guise of using VisiData as a graphing calculator. Which it's not half-bad at!

See the video here: https://www.youtube.com/watch?v=yK3qgOIx4x0

Also, I'm going to be at the PyCon Python conference in a few weeks. If you're going too, send me a note and let's meet up! I'd love to put together a VisiData side session if anyone's interested.

That's all for now, but keep your eyes peeled for v1.2 , which will have (among many other improvements) a 'g+Ctrl-S' command to save all sheets at once.

Bonnes DonnéesVisi, Saul

VisiData Case Study #2: save selected cities from a csv in json format

2018-04-09T00:00:00Z

[originally published as TinyLetter #9]

Hello,

The next installment of the VisiData video series is here! This time, a member of my co-working space had asked if I could create a bubble map of cities from a few particular counties.

It was pretty straightforward to do this with Google Maps and a bit of Javascript, but then I had to get the location and population data for 50 cities into json format. Normally this would require a small script and 20 minutes of trial and error, but fortunately, there is VisiData, and I was able to get this done in only a couple of minutes. Besides being a lot faster, it's also a lot more fun to play with data this way!

So here's the video that explains the process: https://www.youtube.com/watch?v=j0qn8OIiV-w

This video shows how to group rows by multiple columns, and then select rows from particular groups, to be saved into a .json file.

If you have a data task that you'd like to do with VisiData, but aren't exactly sure of the best way to do it, send me an email! There's probably a reasonably easy way to make it happen, and I'd love to help make your data life easier.

Cheers, Saul vd@saul.pw

VisiData Case Study #1 released on Youtube

2018-04-02T00:00:00Z

[originally published as TinyLetter #8]

Hello everyone,

Welcome all you new subscribers! Thank you for your emails, issue submissions, and for spreading the VisiData love around. One of the more frequent comments I've gotten, is that people really enjoy using VisiData, but feel like they're only taking advantage of a small fraction of what it has to offer. So I'm starting to create a new video series to help people get the most out of VisiData.

The first video, released today (no joke!), is a 5 minute case study, which takes a list of donations from a webpage and transforms it into a columnar form that is convertible into USD and easily summed. It was just a trivial curiosity I had a few weeks ago, which would not be worth investigating unless it were as easy as VisiData makes it. But since it was so easy, I thought I would capture the steps and demonstrate how you can replicate the results for yourself.

See the video here: https://youtu.be/yhunJc8Nu4g

In the video, I mention a USD() function which converts arbitrary currency amounts into USD at today's rates. It's not that much code, and I will probably include it in a snippets directory at some point, but if you'd like a private preview, email me and I'll send it to you.

I'm planning to make more of these videos, so also let me know if you have a request for some feature or workflow that you'd like covered!

Happy VisiDating, Saul

VisiData 1.1 released: menus and more

2018-03-06T00:00:00Z

[originally written as TinyLetter #7]

Hello fellow textpunks,

We got a bunch of new subscribers over the weekend, as the lightning talk VisiData video was submitted to HackerNews, and it proceeded to climb onto the front page where it stayed the whole day. This generated a bunch of traffic and a lot of positive response! It seems like people are starting to understand VisiData as more than just a way to view csv files in the terminal.

This was a bit of unfortunate timing, though, since the newcomers all saw v1.0, and we were just about to release v1.1, which has several new features and many improvements. But that's how these things go, and we just kept on our path and released VisiData v1.1 last night as scheduled anyway. The full release notes are on Github (and we'll update the website tonight or tomorrow), but here are the highlights:

The Spacebar now invokes an experimental command menu system. Hopefully this can both let people know what functionality is available, and help them learn the command keys as they explore.
The new ( and ) commands expand and unexpand list/dict columns. This should be very useful with e.g. nested json data.
Data can now be copied directly to the system clipboard. Jeremy Singer-Vine asked for this feature a few months ago, and I can see it being very handy for e.g. pasting a short summary into an email.
Speaking of, Jeremy has been putting together a tutorial for VisiData, which is looking really great! Many people have asked for something like this, and he's done a much better job than I could ever hope to do. Check it out and get some ideas of how to use VisiData to its fullest.
Finally, VisiData has been accepted into Debian! Currently in the unstable repository, it will move into stable for the next Debian release.

I've been thinking a lot lately about the community of terminal users. There are a lot of great tools out there: the old standards like vim, the recent replacements like tmux, modern themes like ohmyzsh; and I hope that someday VisiData will be considered part of that pantheon. Maybe I should convert textpunks.com (which has been sitting idle for a long time) into a lifestyle site for all of us terminal fanatics. Would there be any interest in something like that?

As usual, feedback and suggestions are very welcome; send me an email at vd@saul.pw and say hello!

VisiData 1.0 official release!

2018-01-26T00:00:00Z

[originally published as TinyLetter #6]

Hello again VisiData enthusiasts,

VisiData 1.0 has been officially released! The full release notes (including changelog) can be found at https://github.com/saulpw/visidata/releases/tag/v1.0.

The biggest part of this release is the redesigned and reorganized website, at http://visidata.org. I would love it if you would take a few minutes to fill out a quick survey at https://www.surveymonkey.com/r/8JBN8BM and tell me how you use VisiData and what other features you might want to have.

This is a huge milestone for VisiData. I devoted my sabbatical (the entire year of 2017) to creating an open-source tool that would be useful for myself and other terminal users, and it was very important for me to release a stable and lasting 1.0 so that it could be set aside when I resumed working.

We could have gotten away without certain features and with fewer supported formats, but there is such breadth and depth included in 1.0 (as shown by demos like this one at the PyCascades 2018 conference: https://www.youtube.com/watch?v=N1CBDTgGtOU), that I'm hard-pressed to choose which features should have been omitted.

Now that 1.0 has been released, I would especially love to hear from you. If nothing else, send a quick email to vd@saul.pw and let me know that you're using it!

Happy VisiDating, Saul

VisiData v0.99 released; last stop before 1.0

2017-12-26T00:00:00Z

[originally published as TinyLetter #5]

Hello fellow VisiDataists,

We pushed v0.99 to PyPi last week. This release includes:

tab completion for filenames and expressions;
an html loader;
a json saver;
and several tweaks; see the Changelog for the full list of changes

The only remaining goal for 1.0 is to be installable via a single command on most systems, even if they don't have Python and pip3 already installed.

So we are in the process of undergoing review for inclusion in the Debian package repository and homebrew-core, and hopefully getting close. (The process is more elaborate than it might seem.) Anja has been doing a great job preparing the packages, perfecting the install process, and tying up the all the technical loose ends so that our pull requests won't be rejected. But it is the holidays and these things just take time.

Here's hoping they accept VisiData in the next week or two. But even if they don't, we have our own brew/apt repositories set up, and even now, you should be able to install on MacOS with the single command:

brew install saulpw/vd/visidata

which isn't too bad. But I'd still love to release the formal 1.0 in the official standard repositories.

In the meantime, I would love it if you would test the bejeesus out of v0.99, and file any issues on github. The goal is to have a stable 1.0 that is a reliable workhorse of a tool, that can be useful to myself and others for years to come without any need for major bugfixes or updates.

That is not to say that I'm done with VisiData; in fact, I can't wait to start working on the next version, which will include a more discoverable menu system and some major internal refactoring. But I realize that I've been putting off these refactors for months now because 1.0 seemed closer than it was. So actually getting out a 1.0 stable release would go a long way to ease my mind, before everything gets shuffled around (which might take awhile to become seamless again).

As usual, I would love to hear how you use VisiData. Keep an eye out for the 1.0 release in early January!

Happy holidays, Saul

VisiData v0.98 has been released!

2017-11-24T00:00:00Z

[originally published as TinyLetter #4]

Hello all,

Tonight Anja and I released VisiData v0.98, which can be installed as usual via pip3. In addition to the graphs I demo'ed a few weeks ago, VisiData now also displays maps, sourced from .shp or (vector) .mbtiles. Both graphs and maps are available with the '.' (dot) command (instead of 'm' as mentioned in the preview video). dot == plot.

It also finally has mouse support, which is useful on a canvas to select a cursor region with left click-drag, to scroll around with right click-drag, and to zoom in/out with the scrollwheel. This has not been tested on MacOS, so maybe there is a better binding for e.g. scrolling. If anyone has an opinion on this feel free to weigh in. On a tabular sheet, mouse support is limited to setting the row cursor with left-click and scrolling rows with the scrollwheel.

There are several command and option changes. A few that might affect you:

the 'headerlines' and 'skiplines' options are now called 'header' and 'skip'.
'P' is now "paste before" (like vim). 'R' is now "get a random population sample".
'^Z' now suspends the program. '^O' ("open") launches the external $EDITOR (from the builtin editor).

You can see a longer list of the additions and changes in the release notes (https://github.com/saulpw/visidata/releases/tag/v0.98).

Happy Visidating, Saul

VisiData v0.98 preview: Graphs

2017-11-06T00:00:00Z

[originally published as TinyLetter #3]

Hello VisiDaters,

I recorded a 5 minute screencast to demo the new graphing functionality that will be out in v0.98 later this month. Along the way, I show how to use 'A' and 'ga' to make a new sheet from nothing, and '=' to make computed columns.

Here is the screencast on YouTube: https://youtu.be/Ozap_numsjI

Let me know if you have any workflows or features you are interested in hearing more about!

Saul

VisiData Blog Feed

Estimates of current and future usage

Estimates of current and future usage

Usage Growth

VisiData Turns 5

What is VisiData?

History

2017: pre-v1

2018: v1 era

2019: v1.5.2 era

2020: pre-v2 era

2021: v2 era

tl;dr: the present

The Future of VisiData

Usage Growth

More Usability and Teachability

Bigger Data

Bonkers Plugins

Absurd Amounts of Polish

Here's to another 5 years of VisiData!

VisiData.org Redesign

Goals for the design

Colophon

Challenges overcome

reStructured Text

Documentation search

Updatable components

Design highlights

Command demo

Features and Getting started

Releases and Blog lists on homepage

Before and after

Before:

After:

Issues

10 Ways to Use VisiData

1. Explore datasets effortlessly, no matter the format

2. Convert from any input format into any output format

3. Interactively select replacement for cli tools

4. Scrape HTML table data from a webpage

5. Parse and explore text-based log files

6. Explore a Python object from the REPL

7. Explore any pandas DataFrame

8. Create an adhoc data pipeline

9. rename a bunch of files

10. see all the available options and commands

New feature: Persistent input history

Expansion of CLI Options

The Additions

How Sheet-Specific CLI Options Work

Limitations

VisiData's Limits on Growth

The implications of tabular data architecture

'VisiData is like duct tape for your data', an interview with Saul

Unloaded Sheets

New Features in 2.-3: repeat-last and repeat-input

New feature in 2.-3: Split Window

options.disp_splitwin_pct

Commands

Theme options

Limitations

Hello World!

VisiData Case Study #1 Transcript: summing donation data from a webpage

.visidatarc

Transcription of case study starts

Exploratory Data Analysis Made Easy At The Command Line - Episode 230

Table of Contents

Tobias' intro [00:00]

Saul intro [01:37]

What are the main uses cases for VisiData? [07:22]

The command log [11:48]

How has VisiData evolved over time? [17:12]

Why the terminal? [22:36]

The design of keybindings [29:54]

VisiData's capabilities [36:37]

The community [39:40]

Innovative uses of VisiData [42:51]

The future [46:08]

The terminal renaissance [48:09]

Closing [50:34]

`options.disp_splitwin_pct`

`.visidatarc`