The pen-based interface, called Notate, allows users of computational, digital notebooks to open drawing canvases and handwrite diagrams within lines of traditional, digitised computer code.

This innovation allows far more creativity for programmers, who would usually have to type all of the commands into the code. 

Notate is powered by a deep learning model that has been designed to allow programmers to link handwritten notes and diagrams to textual code and vice versa. For instance, it can identify handwritten programming symbols, like “n”, and link them up to their typewritten equivalents.

“A system like this would be great for data science, specifically with sketching plots and charts that then inter-operate with textual code,” said Ian Arawjo, one of the scientists behind the innovation. 

“Our work shows that the current infrastructure of programming is actually holding us back. People are ready for this type of feature, but developers of interfaces for typing code need to take note of this and support images and graphical interfaces inside code.”

Arawjo also said the work demonstrates a new path forward by introducing artificial intelligence-powered, pen-based coding at a time when drawing tablets are becoming more widely used.

“Tools like Notate are important because they open us up to new ways to think about what programming is, and how different tools and representational practices can change that perspective,” said Tapan Parikh, associate professor of information science and paper co-author.

The scientists’ findings were published in the proceedings of The 35th Annual ACM Symposium on User Interface Software and Technology.

The pen-based interface, called Notate, allows users of computational, digital notebooks to open drawing canvases and handwrite diagrams within lines of traditional, digitised computer code.

This innovation allows far more creativity for programmers, who would usually have to type all of the commands into the code. 

Notate is powered by a deep learning model that has been designed to allow programmers to link handwritten notes and diagrams to textual code and vice versa. For instance, it can identify handwritten programming symbols, like “n”, and link them up to their typewritten equivalents.

“A system like this would be great for data science, specifically with sketching plots and charts that then inter-operate with textual code,” said Ian Arawjo, one of the scientists behind the innovation. 

“Our work shows that the current infrastructure of programming is actually holding us back. People are ready for this type of feature, but developers of interfaces for typing code need to take note of this and support images and graphical interfaces inside code.”

Arawjo also said the work demonstrates a new path forward by introducing artificial intelligence-powered, pen-based coding at a time when drawing tablets are becoming more widely used.

“Tools like Notate are important because they open us up to new ways to think about what programming is, and how different tools and representational practices can change that perspective,” said Tapan Parikh, associate professor of information science and paper co-author.

The scientists’ findings were published in the proceedings of The 35th Annual ACM Symposium on User Interface Software and Technology.